U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences 


Research  Report  1872 


Retention  of  Selected  FBCB2  Operating  Skills  among 
Infantry  Captains  Career  Course  (ICCC)  Students 


Gregory  A.  Goodwin 

U.S.  Army  Research  Institute 

Bruce  C.  Leibrecht,  Richard  L.  Wampler,  Stephen  C.  Livingston 

Northrop  Grumman  Technical  Services 

Jean  L.  Dyer 

U.S.  Army  Research  Institute 


July  2007 


Approved  for  public  release;  distribution  is  unlimited. 


2007081 6064 


U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences 

A  Directorate  of  the  Department  of  the  Army 
Deputy  Chief  of  Staff,  G-1 

Authorized  and  approved  for  distribution: 


MICHELLE  SAMS,  Ph.D. 
DIRECTOR 


PAUL  A.  GADE 

ACTING  TECHNICAL  DIRECTOR 


Research  accomplished  under  contract 
for  the  Department  of  the  Army 

Northrop  Grumman  Technical  Services 

Technical  review  by 

Brooke  B.  Schaab,  U.S.  Army  Research  Institute 
John  S.  Barnett,  U.S.  Army  Research  Institute 


NOTICES 


DISTRIBUTION:  Primary  distribution  of  this  Research  Report  has  been  made  by  ARI. 
Please  address  correspondence  concerning  distribution  of  reports  to:  U.S,  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  Attn:  DAPE-ARI-MS, 

2511  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926 

FINAL  DISPOSITION:  This  Research  Report  may  be  destroyed  when  it  is  no  longer 
needed.  Please  do  not  return  it  to  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 

NOTE:  The  findings  in  this  Research  Report  are  not  to  be  construed  as  an  official 
Department  of  the  Army  position,  unless  so  designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 


1.  REPORT  DATE  (dd-mm-yy) 

July  2007 


2.  REPORT  TYPE 

Final 


4.  TITLE  AND  SUBTITLE 

Retention  of  Selected  FBCB2  Operating  Skills  among  Infantry 
Captains  Career  Course  (ICCC)  Students 


6,  AUTHOR(S)  . 

Gregory  A.  Goodwin  (U.S.  Army  Research  Institute) 

Bruce  C.  Leibrecht,  Richard  L.  Wampler,  and  Stephen  C.  Livingston 
(Northrop  Grumman  Technical  Services) 

Jean  L.  Dyer  (U.S.  Army  Research  Institute) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES)  ~ 
Northrop  Grumman  U.S.  Army  Research  institute  for  the 

Technical  Services  Behavioral  and  Social  Sciences 

3565  Macon  Road  Infantry  Forces  Research  Unit 

Columbus,  GA  31907  POBcx  52086 

Fort  Benning,  GA  31995-2086 


3.  DATES  COVERED  (from  . . .  to) 

August  2005  to  December  2006 


5a,  CONTRACT  OR  GRANT  NUMBER 

W74V8H-04-D-0045  (DO  #08) 


5b.  PROGRAM  ELEMENT  NUMBER 

622785 


5c.  PROJECT  NUMBER 

A790 


5d.  TASK  NUMBER 

326 


5e.  WORK  UNIT  NUMBER 


8.  PERFORMING  ORGANIZATION  REPORT 

NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.  S.  Army  Research  Institute  for  the  Behavioral  &  Social  Sciences 

ATTN:  DAPE-ARI-IJ 

2511  Jefferson  Davis  Highway 

Arlington,  VA  22202-3926 


10.  MONITOR  ACRONYM 

ARI 


11.  MONITOR  REPORT  NUMBER 

Research  Report  1872 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 

Contracting  Officer’s  Representative: 


Jean  L.  Dyer  Subject  Matter  POC:  Gregory  A.  Goodwin 


Vhis“l^CdV"c*reranin"^iga.ion  of  the  retention  of  Force  XXI  Battle  Commend  Brigade  end  (FBCB2) 

nn^ratnr  knowledae  and  skills.  Infantry  captains  who  attended  a  two-day  training  course  participated  i 
P  o  tect  fniinwed  bv  a  retest  eiqht  weeks  later.  Participants  answered  questions  about  and  performed  tasks  on 
FBCB2  Pe“ceyon SI  showed  no  Secay  while  performance  oh  the  hands-on  test  declined 
2(r:=  The  majority  of  participants  (72%)  had  used  FBCB2  in  combat.  Interestingly,  the 

recaU  scores,  implications  of  these  findings  for  trainers,  training  developers,  and  Army  units  are  discussed. 


15.  SUBJECT  TERMS 

Digital  Training  FBCB2  Operator  Skills 
Trainino  Methodology  Performance  Measurement 


Skill  Retention  Knowledge  Retention  Digital  Proficiency 


SECURITY  CLASSIFICATION  OF 

19.  LIMITATION  OF 

ABSTRACT 

20.  NUMBER 

OF  PAGES 

16.  REPORT 
Unclassified 

17.  ABSTRACT 

Unclassified 

18.  THIS  PAGE 

Unclassified 

Unlimited 

21  RESPONSIBLE  PERSON" 

Ellen  Kinzer,  Technical 
Publication  Specialist 
703.602.8047 


l 


. 


11 


Research  Report  1872 


Retention  of  Selected  FBCB2  Operating  Skills  among 
Infantry  Captains  Career  Course  (ICCC)  Students 


Gregory  A.  Goodwin 

U.S.  Army  Research  Institute 


Bruce  C.  Leibrecht,  Richard  L.  Wampler,  Stephen  C.  Livingston 

Northrop  Grumman  Technical  Services 

Jean  L.  Dyer 

U.S.  Army  Research  Institute 


Infantry  Forces  Research  Unit 
Scott  E.  Graham,  Chief 


U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
2511  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926 


July  2007 


Army  Project  Number 
622785A790 


Personnel  Performance 
and  Training  Technology 


Approved  for  public  release!  distribution  is  unlimited. 


til 


The  authors  would  like  to  express  their  gratitude  to  the  digital  instructors  in  the 
Directorate  of  Operations  and  Training  at  Fort  Benning,  Georgia.  Our  data  collection  effort 
would  not  have  been  possible  without  the  cooperation  of  Alfredo  Lezcano,  Scott  Hardcastle, 
Todd  Fancher,  Louis  Leave!!,  and  Nelson  Baldwin.  They  all  contributed  to  this  project  in 
different  ways,  from  reviewing  the  test  materials  to  supporting  data  collection.  Of  a  o  t  lese 
instructors,  Al  Lezcano  deserves  special  mention.  He  served  as  our  primary  point  of  contact 
with  the  instructors  and  generously  contributed  many  hours  of  his  time  helping  to  improve  our 
metrics  refine  our  testing  procedures,  and  answer  our  questions  about  digital  training.  e 
cannot  thank  him  enough  for  his  expertise  and  help.  Additionally  we  would  like  to  thank 
Christopher  Strauss  and  Andy  Perrault  of  Northrop  Grumman  Corporation  for  their  help  in 
developing  the  evaluation  materials,  as  well  as  Mike  Dlubac  and  Amy  Batiz  tor  supporting  t  ae 

collection  of  data. 


iv 


RETENTION  OF  SELECTED  FBCB2  OPERATING  SKILLS  AMONG  INFANTRY 
CAPTAINS  CAREER  COURSE  (ICCC)  STUDENTS 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  Research  Institute’s  Infantry  Forces  Research  Unit  supports  Army 
transformation  by  conducting  research  on  innovative  methods  for  training  o  iers j  an  ®aers. 
Much  of  the  research  focuses  on  building  and  sustaining  proficiency  among  tgl  a  y  etl^'PP 
forces  Sustaining  digital  proficiency  requires  informed  decisions  about  a  oca  mg  irrn 
SSng  resources8 anftime.  Given  the  widespread  use  of  digital  systems,  a  better  ^derstanding 
of  the  decay  of  digital  skills  is  needed  for  training  programs  in  institutions  and  operational  u:  . 
The  current  research  was  deseed  .0  examine  .he  decay  of  skills  and  MW 
familiarization  training  on  the  Force  XXI  Battle  Command  Bngade  and  Below  (FBCB2). 

Procedure: 

The  research  team  administered  tests  of  FBCB2  system  knowledge  and  hands-on  skills  to 
54  career  course  captains  at  Fort  Benning,  Georgia.  The  baseline  test  immediately  followed  a 
two-day  FBCB2  familiarization  course.  The  retest  occurred  eight  wee  s  ater.  a  a  we 
analyzed  to  characterize  the  level  of  mastery  and  subsequent  decay  for  FBCB2  ^owled^and 
skilirParameters.  The  analysis  also  examined  task  dimensions  and  participant  variables  that 

influenced  proficiency  and  retention. 


Findings: 

Overall  scores  on  the  hands-on  test  (72%  at  baseline)  exceeded  those  on  the  knowledge 
test  (40%  correct  at  baseline).  Performance  on  the  hands-on  test  declined  modestly  (10  /.)  a 
ejeht  weeks  whereas  knowledge  recall  was  decay-resistant.  Although  overall  decay  on  the 
hands-on  test  was  small,  greater  decay  was  observed  for  individual  tasks.  Significant WaS 
observed  for  three  of  the  thirteen  hands-on  tasks  with  declines  ranging  from  1 9  /o  to  24  /o. 
Participant  characteristics  (such  as  self-rated  FBCB2  proficiency)  related  to  basic  performance 
parameters  as  well  as  retention  measures.  Multiple  regression  analysis  factoring  in  a  variety  of 
expenence  and  training  measures  accounted  for  25-32%  of  the  variability  in  hands-on  tea  scores. 
An  examination  of  the  specific  errors  made  suggest  that  inadequate  memory  cues  and  inattention 

to  system  details  contributed  to  skill  decay. 

Utilization  and  Dissemination  of  Findings. 

The  findings  and  recommendations  from  this  research  represent  the  first  empirical 
measurements  of  FBCB2  skill  decay.  Although  a  limited  number  of  skills  was  observed,  this 
—  important  because  it  demonstrated  the  utility  of  the  data  collection  techniques  and 
“XT  of  potential  causes  for  skill  decay  on  this  system.  The  findings  can  benefit 
FBCB2  trainers  and  unit  leaders  who  work  to  ensure  Soldier  proficiency  with  digital  skills. 


V 


Findings  could  assist  human  factors  specialists  supporting  system  design.  In  conjunction  with  a 
related  digital  research  effort,  these  findings  were  briefed  to  the  TRADOC  Capabilities  Manager 
Stryker- Bradley  office  at  Fort  Benning,  Georgia. 
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Retention  of  Selected  FBCB2  Operating  Skills  among 
Infantry  Captains  Career  Course  (ICCC)  Students 

Introduction 

The  Army  Battle  Command  System  (ABCS)  is  a  computerized  command  and  control 
(C3)  system.  The  ABCS  family  includes  the  All  Source  Analysis  System  (ASAS)  used  by  the 
intelligence  staff;  the  Advanced  Field  Artillery  Tactical  Data  System  (AFATDS)  used  by  fire 
support  elements;  the  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2)  used  in  vehicles 
and  command  posts  for  tracking  vehicle  locations;  and  the  Maneuver  Control  System  (MCS) 
used  by  the  maneuver  staff. 

These  systems  allow  leaders  to  share  information  including  Global  Positioning  System 
(GPS)  position  information  on  friendly  and  enemy  units,  graphics,  overlays,  reports,  and  orders, 
over  a  tactical  digital  network.  The  information  comes  not  only  from  Soldiers  on  the  battlefield, 
but  from  an  array  of  human  controlled  and  unattended  sensors.  Not  only  do  the  digital  systems 
help  to  distribute  information  but  they  can  be  used  to  analyze  and  manage  it  as  well  (Seacord, 

2000). 

In  theory,  the  ABCS  is  to  be  a  force  multiplier,  allowing  commanders  to  react  faster,  . 
deploy  forces  more  efficiently,  and  ultimately  plan  and  make  decisions  more  rapidly  than  their 
adversaries.  Unfortunately,  the  full  potential  of  these  systems  has  not  been  realized  (Clark, 
2005).  Numerous  problems  including  non-standard  hardware,  software  incompatibilities  across 
different  versions  of  various  systems,  the  need  for  contractor  support  to  maintain  network 
operability,  and  training  that  cannot  keep  pace  with  software  upgrades  have  all  limited  the 
effectiveness  of  these  systems  (Clark,  2005). 

In  addition  to  these  challenges.  Soldiers  and  leaders  have  found  that  their  proficiency  at 
operating  these  systems  is  perishable  if  they  don’t  regularly  work  with  them.  Much  of  the 
evidence  for  this  comes  from  anecdotal  reports  by  various  leaders  (e.g..  Lynch,  2001 )  or  from 
analysis  of  training  exercises  (e.g.,  U.S.  Army  Armor  Center,  1996),  This  belief  was  also 
conveyed  by  leaders  who  participated  in  the  Army  Research  Institute  project  Managing  at  the 
Speed  of  Change  in  Force  XXI  (Johnston,  Leibrecht,  Holder,  Coffey,  &  Quinkert,  2002). 

These  reports  of  the  perishabi  lity  of  digital  skills  agree  with  what  psychologists  have 
known  about  discrete  procedural  skills  (a  category  into  which  most  digital  skills  tall)  since  the 
1950s,  namely  that  they  are  easily  forgotten  relative  to  continuous  skills  (Adams,  1 987). 

Discrete  procedural  skills  are  differentiated  from  continuous  skills  in  that  the  former  have  a 
distinct  beginning  and  end.  Continuous  skills  are  skills  like  riding  a  bike,  which,  as  the 
proverbial  wisdom  goes,  are  rarely  forgotten  once  learned.  In  an  aircraft  cockpit,  for  example, 
pilots  use  checklists  to  avoid  forgetting  discrete  tasks  like  engine  startup  or  takeoff  procedures. 
Studies  have  shown  that  without  rehearsal,  pilots’  memory  for  such  tasks  would  decay  to 
dangerous  levels  within  a  matter  of  weeks.  On  the  other  hand,  pilots'  memory  for  maneuvering 
the  aircraft,  a  continuous  skill,  shows  virtually  no  decay  over  long  periods  of  time  (Schendel, 
Shields,  &  Katz,  1978), 
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Another  type  of  memory  that  is  resistant  to  decay  is  explicit  memory.  Explicit  memory, 
also  called  declarative  memory,  has  been  shown  to  last  for  decades  (e.g.,  Bahrick,  1 979).  Thus, 
one  expectation  of  the  current  research  project  is  that  conceptual  knowledge,  also  called  semantic 
memory  or  common  sense  knowledge,  about  a  digital  system  will  be  more  resistant  to  decay  than 
will  knowledge  for  discrete  procedural  tasks. 

Despite  the  relatively  widespread  belief  that  digital  skills  are  perishable,  there  is  very 
little  experimental  evidence  documenting  either  the  extent  of  the  decay  or  the  specific  tasks  that 
are  most  likely  to  be  forgotten.  In  fact,  Sehaab  and  Moses  (2001 )  found  evidence  that  for  some 
individuals,  there  may  be  little  decay.  In  a  cross-sectional  examination  of  enlisted  military 
intelligence  personnel  who  received  training  on  AS  AS,  these  authors  found  only  a  small  amount 
of  skill  decay.  Specifically,  a  group  of  21  individuals  tested  after  a  two-month  retention  interval 
showed  90%  proficiency  and  two  individuals  scored  78%  on  a  proficiency  test  taken  one  year 
after  training.  The  authors  concluded  that  contrary  to  common  wisdom,  the  digital  skills  they 
observed  showed  very  little  decay  (Sehaab  &  Moses,  2001). 

Larger  amounts  of  skill  decay  were  documented  in  an  experiment  performed  on  the 
retention  of  message  and  overlay  tasks  using  the  In  ter- Vehicular  Information  System  (IVIS),  a 
vehicle  mounted  digital  system  that  pre-dated  FBCB2  (Sanders,  1999).  In  this  experiment. 
Soldiers  were  trained  on  a  series  of  report  and  overlay  tasks.  Overlay  skills  involved  creating 
and  sending  a  graphical  map  overlay  and  report  skills  involved  sending  text  messages.  At  the 
conclusion  of  training,  participants  completed  four  overlay  and  three  report  tasks.  Those 
completing  three  overlay  tasks  successfully  were  considered  overlay  skilled  and  those 
completing  two  report  tasks  were  considered  report  skilled.  Following  a  30  day  interval  with  no 
training,  participants  were  again  given  four  overlay  and  three  report  tasks.  The  researchers  found 
a  significant  (52%)  reduction  in  the  number  of  overlay  skilled  Soldiers  and  a  significant  (23%) 
reduction  in  the  number  of  report  skilled  Soldiers. 

To  date,  no  experimental  data  have  documented  the  rate  of  forgetting  of  tasks  on  the 
current  ABCS  suite  of  systems.  Knowledge  of  the  rates  at  which  various  tasks  are  forgotten  on 
ABCS  systems  is  needed  by  training  developers  and  operational  Army  units.  Training 
developers  need  this  information  to  tailor  training  techniques  and  develop  memory  aids  to 
improve  the  retention  of  the  most  perishable  tasks.  Unit  leadership  needs  this  information  to 
budget  training  time  more  efficiently  by  focusing  on  the  tasks  that  most  need  refreshing. 

To  begin  to  fill  this  gap  in  our  knowledge  of  the  decay  rates  of  digital  tasks,  researchers 
assessed  FBCB2  proficiency  in  a  group  of  officers  attending  the  Infantry  Captains  Career  Course 
(ICCC)  at  Fort  Benning,  Georgia.  The  ICCC  officers  completed  a  knowledge  test  and  a  hands- 
on  proficiency  test  immediately  following  an  FBCB2  training  session  then  repeated  both  tests 
after  an  eight  week  period  with  no  FBCB2  practice. 

This  research  aimed  to  initiate  an  empirical  basis  for  determining  factors  that  impact  the 
retention  of  digital  operating  skills.  The  investigation  focused  on  the  institutional  training 
environment  found  in  schoolhouses  and  installation  learning  centers.  The  specific  research 
objectives  included  developing  and  testing  a  method  for  characterizing  proficiency  of  FBCB2 
knowledge  and  skills  and  capturing  lessons  learned  to  help  improve  retention  of  digital  skills. 
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Method 


Participants 

Participants  were  officers  attending  the  ICCC  at  Fort  Henning,  Georgia  during  the  spring 
and  summer  of  2006.  All  participants  had  just  completed  16  hours  of  FBCB2  familiarization 
training.  During  the  two  days  of  this  familiarization  training,  officers  received  hands-on  training 
of  common  FBCB2  functions  using  a  desktop  computer. 

A  total  of  77  officers  participated  in  the  baseline  measurement  session  and  54  (70%) 
returned  for  the  recall  measurement  session.  For  purposes  of  simplicity  in  data  analysis,  oniy  the 
data  from  the  group  that  participated  in  both  measurement  sessions  were  analyzed.  An 
examination  of  the  23  Captains  who  were  only  tested  at  baseline  revealed  no  distinctive 
characteristics  of  this  group.  For  example,  of  those  participating  in  both  the  baseline  and  recall 
measurements,  80%  had  experience  using  at  least  one  digital  system  in  a  combat  theater 
compared  with  79%  for  those  tested  only  at  baseline.  Of  those  who  participated  in  both 
measurement  sessions,  65%  reported  combat  experience  in  Iraq  compared  with  69%  of  those 
tested  only  at  baseline.  Those  only  tested  at  baseline  were  slightly  less  likely  to  have  combat 
experience  in  Afghanistan  (9%)  than  those  who  participated  in  both  measurement  sessions 
(15%).  Table  1  compares  the  combat  duty  positions  of  the  baseline  only  group  to  the  group  that 
attended  both  baseline  and  recall  sessions.  A  chi-square  comparison  of  these  two  groups  found 
no  significant  differences  across  duty  positions. 


Table  I 

Percent  (%)  of  Participants  in  Various  Duty  Positions  during  Combat  Tours 


Duty  Position 

%  Tested  only  at 
Baseline 
(n  =  23) 

%  Tested  at  Baseline  and 

Recall 
(n  =  54) 

Platoon  Leader 

57 

54 

Operations  Officer  (S3,  AS3,  S3  AIR) 

13 

15 

Executive  Officer 

17 

13 

Logistics  Officer  (S4) 

0 

7 

Battle  Captain 

9 

2 

Liaison  Officer 

0 

2 

Fire  Support  (FDO,  FSO) 

0 

2 

Information  Management  Officer  (S6) 

0 

2 

Comparing  the  baseline  only  group  to  the  participants  who  attended  both  measurement 
sessions  on  other  measures  similarly  failed  to  identify  differences.  Those  tested  only  at  baseline 
were  comparable  to  the  others  in  terms  of  FBCB2  training  and  experience,  self-rated  FBCB2 
proficiency,  general  computer  experience,  and  baseline  performance  on  the  FBCB2  knowledge 
and  hands-on  tests  (see  Table  2).  Comparisons  across  the  two  groups  with  independent  sample  /- 
tests  and  chi-square  analyses  did  not  reveal  significant  differences  for  any  ol  these  measures. 
Thus,  the  factor  or  factors  that  resulted  in  23  individuals  not  returning  for  the  recall  test  did  not 
appear  to  differentially  affect  participants  along  any  of  the  dimensions  measured. 
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Table  2 

Mean  Scores  of  Those  Who  Attended  the  Baseline  Session  Only  versus  Those  Who  Attended  Both 
Sessions 


Measure 

Tested  Only  at 
Baseline 
(n  =  23) 

Tested  at  Baseline 
and  Recall 
(n  =  54) 

Officer  Basic  Course  FBCB2  training  (hours) 

0.6 

0.5 

New  Equipment  Training  (NET)  FBCB2  (hours) 

3.1 

2.7 

FBCB2  use  in  combat  (months) 

8.6 

7.7 

Self-rated  FBCB2  proficiency a 

1.1 

1.2 

Percent  using  any  digital  system  in  combat 

79.0 

80.0 

Composite  computer  experience  b 

5.3 

4.8 

Baseline  knowledge  test  score  c 

8.6 

10.2 

Baseline  hands-on  test  score  d 

22.8 

27.4 

Note.  3  Self-ratings  were  on  a  scale  of  0  -  no  experience,  1  -  basic  experience,  2  -  medium 
experience.  Experience  levels  are  defined  in  the  Measurement  Instruments  section.  b  Composite 
computer  experience  scores  ranged  from  0  to  24.  High  values  reflect  more  computer  experience. 
c  Possible  knowledge  test  scores  ranged  from  0  to  22.  d  Possible  hands-on  test  scores  ranged 
from  0  to  35.  The  computer  experience  scoring,  knowledge  test,  and  hands-on  test  are  described 
in  the  Measurement  Instruments  section. 

Measurement  Instruments 

Participants  completed  three  separate  instruments  in  both  the  baseline  and  recall 
measurement  sessions:  A  questionnaire  regarding  experience  and  training  on  digital  systems,  a 
test  of  knowledge  of  various  FBCB2  functions  and  capabilities,  and  a  hands-on  ability  lest  of  13 
FBCB2  tasks.  These  three  instruments  are  described  in  detail  below,  and  all  the  instruments  used 
in  both  measurement  sessions  can  be  found  in  Appendix  B. 

Experience  and  training  questionnaire.  For  the  baseline  session,  participants  were  asked 
a  set  of  five  questions  regarding  their  experience  and  training  on  ABCS  systems.  In  the  first 
question,  participants  indicated  types  of  individual  operator  training  they  had  received  and  the 
hours  of  instruction  for  each  type  of  training.  Types  of  training  included  online  courses  and  new 
equipment  training.  In  the  second  question,  they  completed  a  checklist  to  indicate  the  types  of 
collective  training  they  had  received  on  ABCS  systems.  Types  of  collective  training  included 
motor  pool  training,  and  various  field  training  or  command  post  exercises  (CPXs).  In  the  third 
question,  participants  listed  the  systems  they  had  used  while  deployed  to  a  combat  theater,  their 
duty  positions  while  using  the  system  and  the  number  of  months  they  used  the  system. 

In  question  four,  participants  rated  their  overall  proficiency  on  ABCS  systems  on  a  four- 
point  scale.  The  levels  of  the  scale  were:  0  -  never  used,  1  -  basic,  2-  medium,  3  -high  ability. 
Nobody  rated  themselves  as  having  high  ability  on  any  of  the  systems  so  the  scores  in  fact  varied 
from  0  to  2.  There  were  behavioral  descriptors  of  each  level  that  reflected  how'  much  ol  the 
system  they  understood,  how  often  they  had  to  ask  for  help,  and  how  often  they  were  asked  for 
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help  by  others  (full  descriptions  of  the  scales  are  in  Appendix  B).  At  a  basic  level,  individuals 
were  saying  that  they  could  use  the  system  to  perform  a  limited  set  of  functions,  At  the  medium 
level  participants  were  saying  that  they  were  knowledgeable  about  most  of  its  functions  and  had 
limited  troubleshooting  experience.  At  the  high  level,  participants  would  be  saying  that  they  had 
advanced  knowledge  of  the  system  and  were  often  asked  for  help  by  others. 

In  the  final  question,  participants  completed  a  checklist  indicating  general  computer 
experience  on  Windows,  Macintosh,  and  Linux  operating  systems.  In  this  checklist,  they 
indicated  whether  or  not  they  had  used  application  software,  installed  software,  installed 
software  patches,  installed  hardware,  changed  boot-up  options  or  Basic  Input  and  Output  System 
(BIOS)  settings,  authored  web  pages  using  software  on  that  system,  or  authored  software  for  that 
operating  system.  To  reduce  the  responses  in  this  checklist  to  a  single  number,  the  total  number 
of  checks  across  all  operating  systems  was  tallied  for  each  participant.  This  score,  composite 
computer  experience,  could  range  from  zero  (indicating  no  experience  on  any  of  the  three 
systems)  to  twenty-four  (indicating  extensive  experience  using  all  operating  systems). 

In  the  recall  session,  the  training  and  experience  questions  were  not  repeated  as  there  was 
no  reason  to  think  that  the  answers  would  have  changed  while  the  students  were  in  the  ICCC  (the 
question  pertained  to  training  received  prior  to  ICCC).  There  was,  however,  the  possibility  that 
they  could  have  used  FBCB2  in  a  training  exercise  during  the  eight  weeks  between  the  two 
testing  sessions,  so  they  were  asked  whether  they  had  used  FBCB2  during  that  time.  If  they 
answered  in  the  affirmative,  they  were  asked  to  list  the  duration  of  operation  in  hours.  None  of 
the  participants  indicated  that  they  had  used  FBCB2  during  the  intervening  period. 

Knowledge  lest.  The  knowledge  test  consisted  of  ten  questions.  The  items  on  this  test 
were  identical  for  the  baseline  and  recall  tests.  Officers  were  asked  to  describe  what  FBCB2 
stands  for,  to  name  the  four  main  areas  of  the  operations  screen,  and  to  explain  how  to  speed  up  a 
system  that  is  running  unusually  slowly  (the  complete  knowledge  test  is  in  Appendix  B).  A 
subject  matter  expert  (SME)  with  experience  training  FBCB2  users  generated  the  questions  and 
answers.  Digital  trainers  at  Fort  Benning,  Georgia  verified  that  all  questions  and  answers  were 
consistent  with  the  material  covered  in  the  course. 

All  of  the  questions  were  fill-in-the-blank  except  one.  Six  of  the  fill-in-the-blank 
questions  had  multiple  components  (e.g.,  “Name  the  2  screens  on  the  FBCB2  system.  ).  In  such 
cases,  officers  were  given  credit  for  each  component  that  was  answered  correctly.  Project 
researchers  developed  the  scoring  criteria  and  tested  them  on  a  randomly  selected  subgroup  of 
tests  from  the  baseline  measurement.  All  knowledge  tests  were  then  scored  independently  by 
two  raters  and  the  responses  were  compared.  When  differences  occurred  between  the  scores  of 
the  two  raters,  they  discussed  those  differences  and  came  to  a  consensus.  The  scoring  criteria 
used  by  the  raters  are  included  in  Appendix  C. 

As  the  completed  tests  were  being  scored,  it  became  apparent  to  the  scorers  that  one  of 
the  ten  items  (#6,  “What  is  meant  by  a  near  real-time  update?”)  was  problematic.  The  course 
instructors  had  covered  this  topic  in  a  more  general  way  than  the  experimenters  had  initially 
understood.  The  end  result  was  that  most  of  the  students  simply  restated  the  question  (e.g.,  “It 
updates  after  a  short  delay”  or  “It  doesn’t  update  in  real-time”).  Applying  the  original  scoring 
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criteria  meant  that  almost  none  of  the  officers  would  have  received  credit,  but  modifying  it  meant 
that  nearly  all  responses  would  receive  credit.  For  this  reason,  we  decided  to  exclude  this  item 
from  further  analyses.  The  remaining  nine  questions  involved  a  total  of  22  separate  responses  on 

the  knowledge  test. 

Hands-on  test.  The  officers  completed  a  13-item,  hands-on  test  in  both  the  baseline  and 
recall  sessions.  Each  officer  used  a  computer  running  the  FBCB2  software  to  complete  the  tasks. 
All  officers  completed  the  hands-on  test  individually.  There  was  no  time  limit,  but  the  average 
time  taken  to  complete  the  hands-on  test  during  the  baseline  test  was  28  min  (range  1 0-44  min) 
and  the  average  time  used  to  complete  the  recall  lest  was  35  min  (range  8-5 1  nun).  Times  were 
recorded  by  the  officers  as  described  in  the  next  section. 

Just  as  with  the  knowledge  test  items,  the  items  on  the  hands-on  test  were  initially 
developed  by  an  SME  with  experience  teaching  FBCB2  to  Soldiers.  The  digital  instructors  at 
Fort  Benning  verified  that  test  items  were  covered  during  the  two-day  training  received  by  the 
ICCC  students.  (The  items  for  both  hands-on  tests  appear  in  Appendix  B.) 

To  score  the  hands-on  test,  a  data  collection  sheet  was  developed  (see  Appendix  C).  This 
form  contained  a  series  of  predominantly  yes/no  questions  for  each  task  on  the  test  (one  question 
for  task  1 1  required  the  experimenter  to  indicate  the  number  of  waypoints  on  the  route).  Scoring 
was  based  on  observable  files,  folders  and  settings  created  by  the  officer  taking  the  test.  For 
example,  in  task  3,  officers  were  asked  to  position  their  icon  on  the  map.  On  the  data  collection 
sheet,  an  experimenter  checked  for  two  things:  a)  whether  the  icon  was  present  on  the  map  and 
b)  whether  the  grid  coordinates  of  the  icon’s  location  were  correct.  Thus,  this  item  had  a 
maximum  score  of  two.  Whenever  an  error  occurred,  a  description  of  the  error  was  recorded  on 
the  data  collection  sheet.  For  example,  if  the  grid  coordinates  were  incorrect  for  task  3,  the 
experimenter  recorded  the  grid  coordinates  that  were  entered  by  the  officer.  All  computers  were 
independently  checked  by  two  data  collectors.  When  there  was  a  disagreement,  both  raters 
looked  at  the  system  and  came  to  a  consensus. 

The  baseline  and  recall  tests  had  the  same  tasks  and  task  order.  However,  some  details  of 
the  tasks  were  altered  on  the  recall  test  to  avoid  a  rote  response;  intent  was  to  evaluate  procedural 
recall,  not  memorization  of  a  specific  answer.  For  example,  in  task  3.  participants  were  asked  to 
manually  place  their  icon  on  the  map.  but  the  coordinates  where  the  icon  was  to  be  placed  varied 
across  the  two  tests.  Other  differences  between  the  tests  included  the  names  of  files  or  folders 
that  officers  were  supposed  to  create  as  well  as  changes  to  some  of  the  settings  they  were  to 
make.  Only  the  wording  for  tasks  1,  2,  and  1 2  remained  completely  unchanged  across  the  two 
tests.  Although  the  wording  for  task  one  was  the  same  on  both  tests  ("What  is  the  role  your 
computer  is  set  to?”),  none  of  the  officers  were  seated  at  the  same  station  during  the  recall  test 
(which  took  place  in  different  classrooms)  so  their  workstation  role  had  changed.  All  tasks  for 
the  baseline  and  recall  hands-on  tests  are  included  in  Appendix  B. 

All  tasks  had  the  same  maximum  score  in  both  the  baseline  and  recall  sessions  with  one 
exception.  Task  nine— create,  save,  and  send  a  SPOT  report— had  a  maximum  score  of  10  for 
the  baseline  session  and  1 2  for  the  recall  session  (officers  had  to  account  for  one  more  enemy 
unit  in  the  recall  session  than  in  the  baseline  session).  1  his  variation  meant  that  there  was  a 
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maximum  score  of  35  for  the  baseline  hands-on  test  and  37  for  the  recall  test.  To  make  the 
results  across  the  two  sessions  directly  comparable,  the  percent  of  the  maximum  score  served  as 
the  measure  of  choice  for  all  items.  During  the  baseline  session,  several  participants  were 
observed  using  the  online  FBCB2  help  function.  We  did  not  anticipate  this  possibility  on  the 
baseline  measurement  so  an  item  was  added  to  the  recall  test  asking  officers  to  indicate  whether 
or  not  they  used  the  online  help  function  for  each  task. 

Procedure 

The  FBCB2  knowledge  and  operator  skill  measurements  were  taken  at  two  points  in 
time.  The  first  measurement  occurred  at  the  conclusion  of  two  days  of  FBCB2  familiarization 
training,  and  the  second  measurement  followed  exactly  eight  weeks  {56  days)  later.  The  two -day 
training  was  part  of  the  normal  ICCC  program  of  instruction.  During  the  FBCB2  training,  the 
instructors  covered  start-up  and  shut-down  procedures,  and  then  all  of  the  major  functions  of  the 
system.  Most  of  the  instruction  followed  a  common  pattern  of  demonstrating  a  procedure  while 
the  students  repeated  the  steps  on  their  own  system,  and  then  having  the  students  complete  a 
practical  exercise  using  that  procedure. 

Testing  took  place  in  digital  classrooms  designed  specifically  for  training  personnel  how' 
to  use  FBCB2.  These  classrooms  were  identical  and  had  about  44  computer  workstations  with 
FBCB2  software  installed.  The  monitors  were  mounted  below  a  plate  glass  desktop  and  were 
facing  upward  so  they  could  be  easily  viewed  through  the  glass  desktop.  This  configuration  also 
#  made  it  difficult  for  officers  to  see  any  adjacent  or  nearby  monitors. 

For  the  baseline  session,  at  the  conclusion  of  their  FBCB2  training,  the  officers  took  a  15- 
min  break.  Upon  returning,  each  participant  received  a  packet  containing  a  privacy  act  statement 
plus  the  three  measurement  instruments — experience  and  training  questionnaire,  knowledge  test, 
and  hands-on  test.  Following  introductory  comments,  participants  were  instructed  to  complete 
the  first  two  of  the  three  instruments  and  then  stop  and  wait  for  further  instructions.  The 
additional  instructions  explained  how  to  record  start  and  stop  times  for  each  hands-on  task. 
Because  all  of  the  officers  executed  the  tasks  simultaneously  at  their  own  pace,  it  was  necessary 
to  have  them  record  their  own  start  and  stop  times.  The  start  time  was  to  be  recorded  after  they 
had  read  the  instructions  but  before  they  began  to  perform  the  task.  They  were  to  record  the  stop 
time  when  they  had  completed  the  task  and  closed  all  dialogue  boxes.  To  illustrate  this 
procedure,  the  proctor  demonstrated  a  task  to  indicate  when  start  and  stop  times  should  be 
recorded.  This  task — using  the  circular  line-of-sight  tool — was  not  part  of  the  hands-on  test. 

A  digital  clock  was  displayed  on  projector  screens  at  the  front  of  the  classroom  and  the 
officers  were  asked  to  record  times  from  that  clock.  Times  were  recorded  to  the  nearest  second. 
If  an  individual  could  not  complete  the  task,  he  was  instructed  to  write  “unable  to  complete”  on 
his  data  sheet.  If  an  individual  forgot  to  record  either  the  stop  or  start  time,  he  was  instructed  to 
go  back  and  repeat  the  task,  recording  those  times  and  indicating  that  they  were  based  on  a 
second  performance  of  the  task.  Officers  were  told  to  proceed  at  their  own  pace  and  to  raise 
their  hands  when  they  completed  all  13  tasks.  An  experimenter  checked  each  officer’s  materials 
to  make  sure  that  they  were  properly  completed,  then  released  the  participant.  On  the  recall  test, 
once  all  officers  were  seated,  the  session  was  conducted  exactly  like  the  baseline  session. 
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After  all  officers  were  released,  two  experimenters  checked  each  machine  using  the 
hands-on  data  collection  instrument  included  in  Appendix  C.  If  there  were  discrepancies 
between  the  data  recorded  by  the  two  experimenters,  they  resolved  those  differences  before 
exiting  the  classroom.  For  the  recall  session,  the  officers  were  told  which  of  two  digital 
classrooms  to  report  to  based  on  an  alphabetical  sorting  of  their  last  names. 

Results 


Experience  and  Training 

In  their  responses  to  the  questionnaire  on  ABCS  training  and  experience,  participants 
indicated  that  they  had  received  a  range  of  both  individual  and  collective  training  experiences 
(see  Tables  3  and  4).  Individual  training  occurred  more  frequently  on  FBCB2  than  the  other 
systems,  although  only  a  minority  of  participants  (less  than  20%)  received  any  given  type  of 
training.  The  bias  for  training  on  FBCB2  probably  reflects  the  fact  that  students  were  infantry 
officers  who  had  led  and  commanded  at  the  platoon  and  company  levels.  The  AS  AS  is  a 
military  intelligence  system  and  AFATDS  is  a  field  artillery  system,  while  MCS  is  found  at  the 
battalion  level  and  above — so  it  is  not  surprising  that  few  if  any  officers  tested  had  prior 
experience  with  any  of  these  systems. 

Nobody  reported  taking  online  training  or  the  Digital  Master  Trainer  course.  Only  1 1% 
of  participants  had  received  new  equipment  training  (NET)  on  FBCB2.  Interestingly,  the  most 
frequent  type  of  individual  training  for  FBCB2  was  “other”  which  was  consistently  defined  by 
participants  as  on-the-job  training  received  while  they  were  deployed  to  Afghanistan  or  Iraq. 


Table  3 

Percent  (%)  of  Participants  Who  Received  Individual  Training  on  Digital  Systems 


Type  of  Training 

None 

Online  Course 

100 

Officer  Basic  Course 

90 

NET 

87 

NET  Delta 

98 

Digital  Master  Trainer 

100 

Other 

82 

FBCB2 

ASAS 

MCS 

AFATDS 

0 

0 

0 

0 

4 

0 

0 

6 

11 

0 

2 

0 

2 

0 

0 

0 

0 

0 

0 

0 

17 

0 

2 

0 

Note.  NET  =  New  Equipment  Training,  NET  Delta  =  NET  on  software  changes  only.  “Other" 
training  occurred  on-the-job  while  deployed  to  Iraq  or  Afghanistan. 


Only  a  minority  of  the  participants  (30%  or  less)  received  collective  unit  training  on  a 
specific  ABCS  system  (see  Table  4).  As  with  individual  training,  FBCB2  was  the  system  for 
which  most  individuals  received  any  collective  training.  Motor  pool  training  with  FBCB2,  the 
most  frequently  reported  type  of  collective  training,  was  claimed  by  only  30%  of  the  participants. 
Field  training  exercises  (FTXs)  and  CPXs  accounted  for  most  of  the  remaining  collective  unit 
training  received  by  the  participants  on  FBCB2. 
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Table  4 

Percent  (%)  of  Participants  Who  Received  Collective  Unit  Training  on  Digital  Systems 


Event 

None 

FBCB2 

Motor  Pool  Training 

65 

30 

FTX  at  Home  Station 

81 

17 

FTX  at  a  Combat  Training  Center 

89 

11 

CPX  at  a  Digital  Training  Facility 

83 

15 

CPX  at  Home  Station 

87 

11 

CPX  at  a  Combat  Training  Center 

98 

0 

Other 

93 

7 

ASAS 


MCS 

4 

2 

2 

0 

0 

0 

4 


AFATDS 

6 

0 

0 

2 

2 

2 

2 


0 

0 

0 

0 

0 

0 

0 


Note.  “Other”  was  consistently  described  as  on-the-job  training  while  deployed  to  Iraq  or 
Afghanistan. 


As  stated  in  the  Methods  section,  80%  of  the  participants  had  experience  using  a  digital 
system  in  combat,  and  most  of  those  gained  their  experience  in  Iraq  (8 1  %)  rather  than 
Afghanistan  (19%).  Nearly  three  quarters  of  the  participants  used  FBCB2  in  a  combat  theater 
(see  Table  5),  while  only  a  small  fraction  (6%  or  less)  used  any  of  the  other  systems  in  combat. 


Table  5 

Combat  Use  of  Digital  Systems  Reported  by  Participants 


Measure 

Any  System 

FBCB2 

ASAS 

MCS 

AFATDS 

%  Who  Used  System  in  Combat 

80 

72 

2 

6 

4 

Average  #  Months  Used 

7.9 

7.9 

10.0 

7.0 

8.0 

Consistent  with  their  training  and  experience,  which  favored  the  FBCB2  system,  8  o  o 
the  participants  rated  themselves  at  either  a  basic  or  medium  level  of  proficiency  on  FBCB2.  In 
contrast,  fewer  than  15%  rated  themselves  at  a  basic  or  medium  level  of  proficiency  on  any  ot 
the  other  systems  (see  Table  6). 


Table  6 

Self  Ratings  of  Operator  Ability  (%  of  Participants) 


Self-Rating 

FBCB2 

ASAS 

MCS 

AFATDS 

Never  Used 

11 

98 

87 

93 

Basic 

59 

2 

9 

1 

A 

Medium 

High 

30 

0 

0 

0 

4 

0 

U 

0 

When  participants  indicated  the  kinds  of  experiences  they  had  with  personal  computers, 
their  responses  reflected  the  prevalence  of  the  Windows  operating  system  on  personal  computers 
(see  Table  7).  Many  participants  indicated  they  had  installed  software,  patches  and  ar  ware  on 
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Windows  systems.  Almost  one  third  had  authored  web  pages  or  changed  boot-up  options  and 
altered  BIOS  settings.  In  contrast,  most  officers  indicated  they  had  no  experience  with  the 
Macintosh  (72%)  or  Linux  (81%)  operating  systems. 


Table  7 

General  Computer  Experience  (%  of  Participants j 


Operating  System 


Activity _ _ 

Never  used  this  operating  system 

Used  application  software 
Installed  application  software 
Installed  software  patches 
Installed  hardware 
Changed  boot-up  options 
Changed  BIOS  settings 

Authored  web  pages  using  software  on  this  system 
Authored  programs  for  this  operating  system 


Windows 

~1T 

94 

81 

56 

48 

31 

30 

31 

17 


Macintosh 

12~ 

2 

0 

0 

0 

0 

0 

0 

0 


Linux 

81 

2 

0 

0 

0 

0 

0 

0 

0 


To  better  understand  how  the  various  training  and  experience  measures  are  related, 
Pearson  correlations  were  calculated  using  several  composite  measures.  These  included  the  sum 
of  all  hours  of  FBCB2  operator  training,  total  number  of  collective  FBCB2  training  experiences, 
total  months  of  FBCB2  use  in  combat,  self-rated  FBCB2  proficiency,  and  composite  computer 
experience.  As  shown  in  Table  8,  self-rated  FBCB2  proficiency  correlated  significantly  and 
positively  with  all  other  training  and  experience  measures  except  for  composite  computer 
experience.  Interestingly,  composite  computer  experience  did  not  correlate  significantly  with 
any  of  the  other  measures  of  training  or  experience. 


Table  8 

Correlations  among  Measures  of  Experience  and  Training 


Measure 

Collective 

FBCB2 

training 

Months  using 
FBCB2  in 
combat 

Self-rated 

FBCB2 

proficiency 

Composite 

computer 

experience 

Total  hours  of  FBCB2  training 

.37  ** 

.24 

.45  ** 

.10 

Collective  FBCB2  training 

.20 

.37  ** 

.10 

Months  using  FBCB2  in  combat 

.50** 

.14 

Self-rated  FBCB2  proficiency 

.18 

**p<.  01 
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Knowledge  Test  Performance  and  Retention 

Overall  performance  analysis.  The  knowledge  test  questions  asked  participants  to  recall 
various  characteristics  of  the  FBCB2  system.  Of  the  nine  questions  scored  (recall  that  question  6 
was  ignored),  six  had  multiple  components  resulting  in  a  total  of  22  separate  responses  on  this 
test.  The  responses  to  the  knowledge  test  were  analyzed  from  the  standpoint  of  (a)  the  total 
number  of  questions  answered  correctly  (i.e.,  all  components  of  a  question  had  to  be  answered 
correctly  to  get  credit  for  a  question)  and  (b)  the  total  number  of  components  answered  correctly. 
There  were  nine  questions  so  the  question  score  could  range  from  0-9;  and  there  were  22 
components  so  the  component  score  could  range  from  0-22. 

The  average  question  score  on  the  baseline  test  was  3.6  (40%  correct)  and  the  average  on 
the  recall  test  was  3.4  (38%  correct).  The  average  component  scores  were  1 0.17  points  (46.2%) 
on  the  baseline  test  and  1 0.24  (46.5%)  on  the  recall  test.  Neither  of  the  changes  was  statistically 

significant.  < 

Table  9  gives  the  correlations  between  measures  of  performance  on  the  knowledge  test. 
Included  are  the  baseline  and  recall  test  scores  for  both  the  component  and  question  measures. 

As  was  expected,  most  of  these  measures  correlated  significantly  and  positively  with  each  other. 
For  example,  the  baseline  question  and  component  scores  correlated  positively  with  the  recall 
question  and  component  scores,  meaning  that  high  recall  scores  were  predicted  by  high  baseline 
scores. 

Table  9 

Correlations  among  Knowledge  Test  Scores 


Question  Score 


Component  Score 
(Recall) 

Baseline 

Recall 

Component  score  (baseline) 

.68** 

.86** 

.45  ** 

Component  score  (recall) 

.58  ** 

.72  ** 

Question  score  (baseline) 

.42  ** 

**  p  <  .01. 


hem  analysis.  Despite  the  negligible  forgetting  overall,  there  were  some  sizable  changes 
in  performance  on  individual  questions.  Table  10  presents  the  percent  of  the  sample  that 
correctly  answered  each  question  on  the  knowledge  test. 

Chi-square  tests  were  performed  to  compare  the  proportion  that  answered  each  question 
correctly  on  the  recall  test  as  compared  to  the  baseline  test,  using  McNemar’s  (1975)  method  tor 
correlated  proportions.  There  was  significant  forgetting  on  only  one  of  the  questions,  #  5, 
Interestingly,  there  was  a  significant  improvement  in  performance  on  two  ot  the  questions,  #7 
and  #1 .  The  chi-square  values  and  exact  alpha  levels  are  provided  in  Appendix  D. 
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Table  10 

Percent  (%)  of  the  Sample  Answering  Each  Question  Correctly  on  the  Knowledge  Test 


Question 

Baseline 

Recall 

Difference 

5.  Packet  mode  message  size  limit? 

69 

24** 

-45 

3.  The  two  system  screens  are? 

15 

4 

-11 

2.  FIPR  stands  for  what? 

46 

39 

-7 

10.  Time  zone  to  enter  reminders? 

74 

70 

-4 

9.  Advantage  of  FBCB2  and  BFT? 

0 

0 

0 

4.  Four  main  areas  of  Ops  screen? 

0 

2 

2 

8.  How  to  speed  up  a  slow  system? 

67 

76 

9 

7.  LOS  tool  determines  what? 

48 

67* 

19 

1 .  FBCB2  stands  for  what? 

43 

63* 

20 

Overall 

40 

38 

-2 

Note:  FIPR  =  Flash,  Immediate,  Priority,  Routine  (categories  of  message  importance),  BFT  = 
Blue  Force  Tracking,  LOS  =  line  of  sight. 

**  p<  .01,  indicating  that  more  individuals  decreased  than  increased.  *  p  <  .05,  indicating  that 
more  individuals  increased  than  decreased. 

Table  !  1  presents  the  percent  of  the  sample  correctly  answering  the  components  of  each 
question.  McNemar  chi-square  analyses  were  performed  on  the  6  questions  that  had  multiple 
components  to  highlight  significant  changes  across  the  two  tests  (see  Appendix  D).  Question  5, 
the  only  question  on  which  performance  declined  significantly,  had  only  one  component,  so  it  is 
not  included  in  Table  1 1 .  Interestingly,  for  question  #7,  although  the  question  performance 
improved  significantly,  performance  did  not  improve  significantly  on  either  of  the  components. 
This  suggests  that  a  similar  number  of  errors  was  made  on  the  baseline  and  recall  tests  but  those 
errors  were  made  by  fewer  individuals  on  the  retest.  For  question  #1,  the  improvement  in  recall 
for  the  acronym  “FBCB2”  was  driven  largely  by  an  improvement  in  recall  of  “Force  XXI”  and 
“brigade  and  below”  although  recall  for  all  letters  of  the  acronym  did  increase  across  the  two 
tests.  For  question  #4  (list  the  four  main  areas  of  the  operations  screen),  performance  did  not 
change  on  the  question  score  but  there  was  a  decline  in  the  component  scores.  Specifically,  there 
was  a  significant  decline  in  remembering  the  classification  banner  and  operations  function  bar. 

An  examination  of  the  errors  made  on  the  recall  test  is  useful  for  understanding  what  wras 
typically  forgotten.  For  example,  in  questions  1  and  2,  participants  were  asked  to  spell  out 
acronyms  (FBCB2  and  FIPR,  respectively).  Recall  for  “FBCB2”  improved  significantly  while 
recall  for  “FIPR”  declined  slightly.  Those  who  made  mistakes  on  question  1  (FBCB2)  often 
only  missed  one  word  (8  individuals).  The  mistakes  were  often  not  too  different  from  the  correct 
word  (e.g.,  battlefield  in  lieu  of  battle,  or  communications  in  lieu  of  command).  Everyone  who 
attempted  to  answer  got  at  least  one  word  correct  and  only  seven  gave  no  response.  In  contrast, 
when  answering  question  2  (FIPR)  participants  either  knew  the  answer  or  they  didn't.  Only  1 0 
individuals  received  partial  credit.  The  remaining  44  were  split  between  full  credit  (21 
participants)  and  no  credit  (23  participants). 


12 


Table  11 

Percent  (%)  of  the  Sample  Answering  Each  Component  Correctly  on  the  Knowledge  Test 


Question  and  Components 

Baseline 

Recall 

Difference 

3.  Name  the  two  screens  on  FBCB2. 

Session  manager 

15 

6 

-9 

Operations 

41 

28 

-13 

2,  What  does  FIPR  stand  for? 

Flash 

48 

41 

-7 

Immediate 

48 

44 

-4 

Priority 

59 

57 

-2 

Routine 

46 

44 

-2 

9.  List  one  advantage  to  FBCB2  and  BFT. 

FBCB2 

0 

0 

0 

BFT 

20 

19 

-1 

4.  What  are  the  four  main  areas  of  the  Ops  screen? 

Classification  banner 

20 

7  * 

-13 

Map  display 

39 

46 

7 

Operations  function  bar 

19 

4  * 

-15 

Communications/FIPR  queue 

11 

7 

-4 

7.  The  LOS  tool  allows  the  Soldier  to  determine? 

Bearing 

56 

72 

16 

Range 

69 

83 

14 

1 .  What  does  FBCB2  stand  for? 

Force  XXI 

56 

76** 

20 

Battle 

67 

78 

11 

Command 

70 

80 

10 

Brigade  and 

65 

83  * 

18 

Below 

59 

76  * 

17 

* p<  .05,  ** p  <  .01  indicating  that  the  proportion  answering  correctly  on  the  recall  test  was 
significantly  different  from  the  proportion  answering  correctly  on  the  baseline. 


On  the  recall  test  for  question  3,  name  the  two  FBCB2  screens,  about  half  of  those  who 
got  no  credit  (21  of  39  participants)  did  not  even  venture  a  guess,  indicating  that  they  had  no 
recollection  of  the  answer.  Participants  were  more  likely  to  recall  the  operations  screen  than  the 
session  manager  screen  which  was  not  surprising  as  most  work  on  FBCB2  is  done  on  the  former 
screen.  The  most  common  mistakes  naming  this  screen  involved  assigning  names  related  to  its 
function,  such  as  "map",  "maneuver",  or  "navigation." 

When  asked  to  name  the  four  main  areas  of  the  operations  screen  in  question  4  of  the 
recall  test,  participants  clearly  struggled.  About  half  of  the  participants  (25  individuals)  recalled 
that  the  map  was  one  area,  but  fewer  than  five  were  able  to  recall  any  of  the  other  areas.  A 
number  of  answers  indicated  that  they  recalled  the  function  of  the  area  though  they  could  not 
recall  the  exact  name  of  the  area.  For  example,  14  individuals  listed  “message  line”  or  “message 
bar”  instead  of  FIPR  queue.  Several  also  mentioned  a  “toolbar”  or  “tools”  by  which  they 
probably  meant  the  function  bar. 
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Participants  performed  at  a  chance  level  on  question  5,  the  only  multiple  choice  question 
of  the  test  (24%  correct  on  a  four-choice  question).  For  some  reason,  the  most  popular  selection 
was  1280  KB  (39%  of  the  sample).  This  may  indicate  a  guessing  strategy  of  avoiding  the 
extremes  as  it  was  the  second  largest  value  among  the  four  choices  (the  correct  answer  was  1450 
KB,  the  largest  value).  There  was  nothing  intuitive  about  this  question  and  to  answer  correctly, 
participants  would  simply  have  to  recall  the  correct  figure. 

Questions  7  and  8  on  the  other  hand  were  both  relatively  intuitive  for  someone  with  a 
military  background,  and  consequently  participants  did  well  on  them.  Question  7  asked  them  to 
list  information  provided  by  the  LOS  tool.  Only  three  individuals  left  the  question  blank,  and 
only  three  who  attempted  to  answer  received  no  credit.  When  asked  how  to  speed  up  a  slow 
system  in  question  8,  only  one  person  did  not  respond;  of  the  thirteen  with  incorrect  responses, 
seven  said  either  restart  or  reboot  the  system.  These  were  not  given  credit  because  clearing  logs 
and  queues  is  not  really  a  system  reboot.  Furthermore,  this  response  may  have  been  a  guess  as 
rebooting  a  Windows  system  is  a  common  remedy  for  a  computer  that  is  running  slower  than  it 
should. 


In  question  9,  list  one  advantage  of  FBCB2  and  BFT,  few  participants  answered 
correctly.  Twenty-nine  (54%)  of  the  participants  said  they  did  not  know  the  answer  or  left  at 
least  one  of  the  parts  blank.  None  of  the  participants  knew  an  advantage  for  FBCB2  (larger 
bandwidth)  although  some  said  that  it  was  faster.  Ten  participants  knew  an  advantage  of  BFT 
(signal  is  not  impeded  by  terrain).  Most  of  the  incorrect  responses  represented  general 
advantages  of  both  systems  (e.g.,  track  friendly  forces,  avoid  fratricide)  but  failed  to  note 
advantages  of  one  system  over  the  other. 

Finally,  in  question  10,  participants  were  asked  to  indicate  what  time  zone  must  be  used 
for  periodic  reminders.  Most  of  the  participants  recalled  that  it  was  Zulu  or  Greenwich  Mean 
Time  but  16  did  not.  Of  those  16,  13  responded  with  “local  time.”  It  is  not  clear  why  there  was 
such  consistency  among  those  who  got  this  question  wTong  other  than  it  seemed  a  logical  guess. 

Hands-on  Test  Performance  and  Retention 

Overall  performance  analysis.  The  hands-on  test  entailed  13  tasks  (see  Appendix  B) 
performed  on  an  FBCB2  workstation.  Scoring  of  each  task  depended  on  the  observable  footprint 
left  on  the  FBCB2  workstation  (e.g.,  checking  to  see  that  an  icon  was  in  the  right  location).  As 
with  the  knowledge  test,  a  task  score  (analogous  to  the  knowledge  test  question  score)  reflected 
the  number  of  tasks  for  which  all  components  were  performed  correctly.  Hands-on  task  scores 
ranged  from  0-13.  In  addition,  a  hands-on  component  score  (analogous  to  the  knowledge  test 
component  score)  was  derived  to  reflect  the  number  of  task  components  completed  correctly. 
There  were  a  total  of  35  components  on  the  baseline  test  and  37  components  on  the  recall  test. 
Because  the  number  of  components  varied  across  the  two  tests,  the  analysis  was  done  on  the 
percent  of  components  performed  correctly. 

As  seen  in  Figure  1,  performance  summed  across  the  13  tasks  showed  modest  (10%) 
decay.  In  the  baseline,  the  participants  completed  an  average  of  72%  of  the  tasks  correctly 
without  errors  in  any  of  the  components  and  this  dropped  to  62%  in  the  recall  test.  Analysis  of 
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variance  revealed  the  test-retest  effect  was  significant,  F(l,53)  =  17.07,/?  <  .001.  When  looking 
at  component  scores,  less  decay  was  seen.  Component  scores  averaged  78%  correct  in  the 
baseline  test  and  73%  correct  in  the  retest.  The  modest  (5%)  decrease  was  statistically 
significant,  .F(1,53)  =  4.09,  p  <  .05. 


Figure  1.  Mean  scores  on  the  hands-on  baseline  and  recall  tests. 

Table  12  presents  the  correlations  among  the  overall  component  and  task  scores  on  the 
baseline  and  recall  test  scores.  As  with  the  knowledge  test,  there  were  significant,  positive 
correlations  among  the  baseline  and  recall  tests.  This  indicates  that  high  scores  at  baseline  were 
related  to  higher  scores  at  recall.  The  highest  correlations  were  between  the  component  and  task 
scores  at  baseline  and  similarly  between  these  two  scores  at  recall. 

Table  12 

Correlations  among  Hands-on  Test  Scores 


Task  Score 


Component  Score 
(Recall) 

Baseline 

Recall 

Component  score  (baseline) 

.45  ** 

.85  ** 

.60  ** 

Component  score  (recall) 

.47** 

.92  ** 

Task  score  (baseline) 

.61  ** 

**/><. 01. 
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Item  analysis.  Table  1 3  shows  the  percent  of  the  sample  correctly  performing  each  of  the 
13  tasks.  To  compare  performance  across  the  two  testing  periods,  McNemar  chi-square  analyses 
were  performed  on  the  recall  data  for  each  task  (see  Appendix  D  for  chi-square  statistics).  Chi- 
square  results  indicated  that  there  was  a  significant  decline  in  performance  on  three  of  the 
thirteen  tasks.  The  tasks  arc  arranged  in  Table  13  with  those  showing  the  greatest  loss  at  the  top. 
Compared  to  the  mean  decline  of  1 0%  across  all  tasks,  the  tasks  showing  a  significant  decline 
exceeded  this  value  by  a  factor  of  two  or  more. 


Table  13 

Percent  (%)  of  the  Sample  Correctly  Performing  the  13  Hands-on  Tasks 


Task 

Baseline 

Recall 

Difference 

12.  Position  and  center  icon  on  map3 

87 

63  ** 

-24 

1 1 .  Create  route  on  map 

46 

26  * 

-20 

6,  Create  address  group 

74 

55  * 

-19 

2.  Clear  logs  and  queues 

33 

18 

-15 

7.  Set  free  text  defaults 

65 

52 

-13 

3.  Position  icon  on  map 

76 

63 

-13 

10.  Save  incoming  message 

91 

78 

-13 

4.  Adjust  SA  settings 

89 

78 

-11 

5.  Create  message  folders 

100 

91 

-9 

I .  Verify  platform  role 

93 

91 

-2 

8.  Create/send  free  text  message 

85 

85 

0 

9.  Create/send  SPOT  report 

22 

24 

2 

13.  Check  line  of  sight 

69 

72 

3 

3  Task  12  was  not  independent  of  task  3  because  to  center  the  icon  on  the  map,  it  must  first  be 
placed  on  the  map. 

*  p<  .05,  *♦/?<. 01. 


It  is  important  to  note  that  task  12,  center  the  icon  on  the  map,  could  not  be  executed 
unless  part  of  task  3,  position  the  icon  on  the  map,  was  done  successfully.  The  data  lor  task  12 
in  Table  13  reflect  this  dependency,  showing  that  the  percent  of  participants  who  accomplished 
both  tasks  declined  significantly.  If  only  those  people  who  succeeded  in  placing  their  icon  on  the 
map  (whether  or  not  it  was  in  the  correct  location)  are  examined,  there  is  only  a  small,  non¬ 
significant  decline  in  their  ability  to  also  center  the  icon  (90%  on  the  baseline  test  vs.  87%  on  the 
recall  test).  Thus  the  significant  decline  in  performance  on  task  12  is  primarily  due  to  forgetting 
how  to  get  the  icon  on  the  map,  not  how  to  center  the  icon. 

To  better  understand  exactly  which  component  of  each  of  the  hands-on  tasks  contributed 
most  to  forgetting,  the  performance  on  each  component  of  every  task  was  examined  (see  Table 
14).  In  cases  where  there  was  only  one  component  for  a  task,  the  percentages  are  the  same  as 
those  show'n  in  Table  13,  therefore  they  are  not  repeated  in  Table  14.  Of  the  33  components 
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analyzed,  four  showed  a  significant  decline  and  one  showed  a  significant  increase  (see  also 
Appendix  D). 

Table  14 

Percent  (%)  of  the  Sample  Correctly  Completing  Components  of  Each  Hands-on  Task 


Task  and  Components  Checked 

1  ] .  Create  route 

Route  displayed  on  screen 

Route  named  correctly 

Route  number  of  waypoints  correct 

6.  Create  address  group 
Address  group  named 
Address  group  addressee  correct 

2.  Clear  logs  and  queues 

One  message  in  FIPR  queue 
No  old  messages 
No  other  directories 

3.  Position  icon  on  map 
Icon  visible  on  map 
Icon  location  correct 

7.  Set  free  text  message  defaults 

Free  text  message  default  address  set 
Free  text  message  default  precedence  set 
Free  text  message  default  acknowledge  set 

4.  Adjust  SA  settings 

SA:  stale  setting  correct 
SA:  old  setting  correct 
SA:  purge  setting  correct 

5.  Create  message  folders 
Directory  1  named 
Directory  2  named 

8.  Create/send  free  text  message 
Free  text  message  sent 

Free  text  message  addressee  correct 

9.  Create/save/send  SPOT  report 
SPOT  report  in  folder 
SPOT  report  named  correctly 
SPOT:  entity  1  indicated 
SPOT:  entity  1  location  indicated 
SPOT:  entity  I  status  indicated 
SPOT:  entity  1  activity  indicated 
SPOT:  entity  2  indicated 
SPOT:  entity  2  speed  indicated  a 
SPOT:  entity  2  direction  indicated  8 
SPOT:  precedence  set 

SPOT:  report  sent 
SPOT:  addressee  correct 


Baseline 

Recall 

Difference 

72 

41  ** 

-31 

93 

83 

-10 

57 

48 

-9 

82 

72 

-10 

74 

56* 

-18 

72 

67 

-5 

80 

78 

-2 

41 

30 

-11 

100 

72* 

-28 

76 

63 

-13 

74 

54  * 

-20 

78 

76 

-2 

80 

76 

-4 

89 

80 

-9 

89 

83 

-6 

89 

82 

-7 

100 

94 

-6 

100 

91 

-9 

93 

91 

-2 

85 

85 

0 

82 

89 

7 

54 

48 

-6 

82 

83 

1 

63 

82  * 

19 

83 

89 

6 

82 

82 

0 

82 

82 

0 

82 

82 

48 

41 

-7 

85 

72 

-13 

48 

59 

11 
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Table  14  (continued) 


Task  and  Components  Checked 

Baseline 

Recall 

Difference 

13.  Check  line  of  sight 

0 

LOS  trace  displayed 

70 

72 

2 

LOS  height  correct 

74 

80 

6 

Note.  Tasks  1,10,  and  12  each  had  only  one  component  so  they  are  not  repeated  in  this  table. 
a  These  items  were  present  in  only  the  retest,  not  the  baseline. 

*p  <  .05,  **  p<  .01. 


Not  all  components  of  each  task  were  forgotten  equally.  For  example,  performance  on 
task  1 1  declined  significantly  overall,  but  this  decline  was  primarily  due  to  participants  forgetting 
how  to  keep  the  route  displayed  on  the  map.  Participants  did  not  forget  how  to  name  the  route  or 
add  the  correct  number  of  waypoints  (Table  14).  For  task  6,  created  address  group,  individuals 
remembered  how  to  name  the  address  group  but  they  forgot  how  to  assign  the  correct  addressee. 

There  were  significant  changes  in  performance  on  components  of  two  other  tasks  even 
though  there  was  no  significant  decay  on  the  overall  task  scores.  For  task  3,  position  icon  on 
map,  there  was  significant  decay  for  putting  the  icon  on  the  map  but  no  decay  for  putting  it  in  the 
correct  location.  On  task  7,  set  free  text  message  defaults,  participants  forgot  how  to  set  the 
default  addressee  but  not  how  to  change  the  precedence  and  acknowledgement  settings.  On  task 
9  (create,  save,  and  send  a  SPOT  report),  participants  actually  improved  their  ability  to  indicate 
the  location  of  the  first  entity,  but  showed  no  changes  in  performance  on  any  of  the  other 
components. 

Common  Errors  on  the  Recall  Test 

As  writh  the  knowledge  test,  an  examination  of  the  incorrect  responses  to  the  hands-on 
tasks  sheds  some  light  on  the  nature  of  what  was  forgotten.  For  some  of  components  of  the 
hands  on  tasks,  it  was  only  possible  to  know  whether  or  not  the  procedural  step  was  completed. 
However,  for  several  of  the  components,  it  was  possible  to  examine  the  nature  of  the  mistakes 
that  were  made.  What  follows  is  a  discussion  of  the  types  of  errors  made  on  the  components  of 
the  hands-on  tasks  in  the  recall  test.  These  errors  are  also  summarized  in  Table  15. 

On  the  first  task,  copying  the  assigned  role  of  the  system,  almost  everyone  was  able  to 
perform  this  task.  Judging  by  the  responses  of  the  four  individuals  who  did  not  get  credit  for  this 
task,  they  did  not  appear  to  know  what  they  were  looking  for  since  they  entered  various  phrases 
or  labels  on  the  screen  (e.g.,  “NAV”,  or  “Commsat:  GO”)  or  they  attempted  to  copy  the  roles 
from  nearby  participants. 

On  task  2,  clear  logs  and  queues,  44  of  54  (81  %)  were  unable  to  perform  this  task.  There 
was  no  way  to  determine  which  step  in  clearing  logs  and  queues  was  most  easily  forgotten,  but  it 
is  interesting  to  note  that  more  individuals  used  help  on  this  task  than  on  any  other  task  (see 
Table  1 6).  Only  34  of  54  (63%)  of  the  participants  were  able  to  place  their  icon  on  the  correct 
map  location  for  task  3.  Of  the  20  who  did  not  get  credit  on  this  task,  14  were  not  able  to  place 
their  icon  on  the  map  at  all.  The  remaining  six  individuals  were  able  to  place  their  icon  on  the 
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map  but  put  it  in  the  wrong  location,  most  likely  because  they  simply  clicked  on  the  map  rather 
than  entering  the  exact  coordinates. 


Table  15 

Rates  for  Types  of  Errors  Made  on  the  Hands-On  Recall  Test 


Task  and  Error  Description 

# 

making 

errors 

Percent 

of 

sample 

Percent  of 

those 

making 

errors 

1 .  Verify  platform  role 

5 

9% 

did  not  know  what  to  look  for 

4 

7% 

80% 

transcription  errors 

1 

2% 

20% 

4.  Adjust  SA  settings 

12 

22% 

changed  settings  with  errors 

3 

6% 

25% 

did  not  change  from  default 

9 

17% 

75% 

5.  Create  two  message  folders 

5 

9% 

created  only  one  folder 

2 

4% 

40% 

did  not  create  any  folders 

3 

6% 

60% 

6.  Create  address  group 

24 

44% 

did  not  create  address  group 

14 

26% 

58% 

wrong  addressee  due  to  failure  to  check  search  result 

8 

15% 

33% 

wrong  addressee 

2 

4% 

8% 

address  group  named  incorrectly 

1 

2% 

4% 

7.  Set  free  text  message  defaults 

26 

48% 

no  addressee 

11 

20% 

42% 

precedence  left  at  default 

10 

19% 

38% 

wrong  addressee  due  to  failure  to  check  search  result 

10 

19% 

38% 

precedence  incorrect  (not  default) 

3 

6% 

12% 

self  addressed 

2 

4% 

8% 

8.  Create/send  free  text  message 

8 

15% 

wrong  addressee  due  to  failure  to  check  search  result 

8 

15% 

100% 

failed  to  send 

5 

9% 

63% 

9.  Create/send  SPOT  report 

41 

76% 

precedence  left  at  default 

25 

46% 

61% 

message  name  left  at  default 

19 

35% 

46% 

no  addressee/not  sent 

10 

19% 

24% 

message  named  incorrectly  (not  default  name) 

9 

17% 

22% 

wrong  addressee  due  to  failure  to  check  search  result 

8 

15% 

20% 

precedence  incorrect  (not  default) 

7 

13% 

17% 

no  message  created 

3 

6% 

7% 

self  addressed 

1 

2% 

2% 

10.  Save  incoming  message 

12 

22% 

saved  in  wrong  location 

4 

7% 

33% 

did  not  save  message 

8 

15% 

67% 

19 


Table  15  (continued) 


Task  and  Error  Description 

# 

making 

errors 

Percent 

of 

sample 

Percent  of 
those 
making 
errors 

1 1 .  Create  route  on  map 

40 

74% 

route  was  not  displayed 

32 

59% 

80% 

route  had  more  than  5  waypoints 

21 

39% 

52% 

no  route  was  created 

7 

13% 

18% 

route  had  default  name 

2 

4% 

5% 

Note.  To  avoid  redundancy  with  Table  14,  information  is  provided  only  for  those  tasks  in  which 
information  about  the  nature  of  the  errors  is  available. 


Most  participants  were  able  to  perform  tasks  4  and  5.  When  asked  to  change  the  situation 
awareness  settings  in  task  4,  nine  individuals  were  completely  unable  to  perform  the  task  (with 
settings  remaining  at  default  values)  and  three  individuals  changed  the  settings,  demonstrating 
they  knew  what  to  do,  but  they  made  errors  in  their  changes.  Most  everyone  was  able  to  perform 
task  5,  create  two  message  folders.  Only  three  participants  could  not  create  any  folders,  and  two 
created  one  folder  demonstrating  they  knew  the  process. 

In  tasks  6,  7,  8,  and  9.  participants  were  asked  to  create  and/or  address  various  types  of 
messages.  An  error  common  to  all  four  of  these  tasks  stemmed  from  using  the  search  function  to 
find  an  addressee.  The  address  book  search  function  in  FBCB2  does  not  always  return  an  exact 
match  to  the  search  string,  even  when  one  exists.  For  this  reason,  it  is  always  important  to 
double  check  the  search  result  before  selecting  the  addressee.  For  tasks  6-9,  using  the  address 
book  search  function  returned  a  close,  but  incorrect,  addressee  (deputy  commander  vs. 
commander).  Participants  who  did  not  check  the  search  result  before  selecting  it  made  errors. 

On  all  of  these  tasks,  20-100%  of  those  who  made  errors  misaddressed  their  messages  because 
they  did  not  carefully  check  the  search  result  before  selecting  the  addressee.  Otherwise, 
assigning  an  address  seemed  to  create  problems  for  tasks  6-9.  Often,  participants  were  not  able 
to  assign  an  address  at  all,  or  wound  up  self-addressing  the  message. 

In  tasks  7  and  9,  participants  had  to  alter  the  precedence  settings  of  the  outgoing 
messages  and  this  was  the  most  common  source  of  error  for  task  9.  In  task  7,  half  of  those  who 
made  errors  failed  to  set  the  precedence  of  the  message  correctly,  and  in  task  9  over  two-thirds  of 
those  making  errors  assigned  the  wrong  precedence.  Most  commonly,  participants  left  the 
precedence  setting  at  its  default  value,  indicating  that  they  had  forgotten  how  to  change  it. 

Fewer  participants  had  trouble  with  tasks  10  and  1 1.  For  task  10,  display  and  save  a 
message,  only  eight  individuals  were  unable  to  save  the  message  and  four  individuals  saved  it  to 
the  wrong  location.  For  task  1 1,  create  a  route,  only  seven  individuals  completely  failed  to 
create  a  route.  The  most  common  error  on  task  1 1  (made  by  25  individuals)  was  that  participants 
who  created  a  route,  failed  to  leave  the  route  displayed.  Another  common  error  for  task  1 1  was 
that  too  many  waypoints  were  added  to  the  route.  Participants  were  instructed  to  create  a  route 
with  five  w'aypoints.  Most  commonly,  participants  generated  six  waypoints,  an  error  that 
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probably  resulted  from  failing  to  count  their  current  location  as  a  waypoint.  Of  those  who  made 
errors  counting  waypoints,  1 3  had  six  waypoints,  five  had  seven,  two  had  eight,  and  one  had  3 1 . 

There  were  no  insights  regarding  the  errors  made  on  components  of  tasks  12  and  13 
because  we  were  only  able  to  ascertain  whether  or  not  participants  completed  those  components. 

Time  to  Complete  Hands-on  Tasks 

In  the  baseline  test,  the  participants  took  an  average  of  28.0  min  total  time  to  perform  the 
hands-on  test.  The  total  elapsed  time  was  computed  by  subtracting  the  Task  1  start  time  from  the 
Task  1 3  end  time,  so  it  included  the  time  spent  reading  directions  and  executing  the  tasks.  The 
participants  expended  an  average  of  35. 1  min  in  performing  the  recall  test,  for  a  mean  increase  of 
7. 1  min.  The  increase  in  time  was  significant,  /(50)  =  6.47,  p  <  .001 .  [Note:  Three  participants 
had  missing  data  in  the  baseline  or  recall  test.] 

For  the  thirteen  tasks  of  the  hands-on  test,  the  average  execution  times  appear  in  Table 
1 6.  For  each  task  the  means  are  based  on  all  participants  with  valid  times  for  both  the  baseline 
and  recall  tests  (excluding  outliers).  Participants  with  valid  times  were  included  whether  the 
recorded  time  was  associated  with  correct,  partially  correct,  incorrect,  or  incomplete 
performance.  Data  were  analyzed  in  a  series  of  repeated -measures  /-tests  to  compare  the  time 
taken  to  complete  each  task  in  the  baseline  and  recall  tests.  Table  16  places  the  tasks  showing 
the  greatest  increase  at  the  top. 

Table  16 

Mean  Time  Spent  per  Task  in  Hands-on  Tests  (Baseline  and  Recall) 


Task 

n 

Baseline  Mean 
(min) 

Recall  Mean 
(min) 

Difference 

(min) 

3.  Position  icon  on  map 

48 

1.55 

3.02  ** 

1.47 

2.  Clear  logs  and  queues  a 

47 

3.77 

5.07  ** 

1.30 

6.  Create  address  group 

41 

1.83 

2.57  ** 

0.74 

4.  Adjust  SA  settings 

53 

0.85 

1.48  ** 

0.63 

9.  Create/send  SPOT  report h 

47 

3.63 

4.17 

0.54 

5.  Create  message  folders 

53 

1.17 

1.68  ** 

0.51 

1 2.  Center  icon  on  map 

45 

0.11 

0.41  ** 

0.30 

10.  Save  incoming  message 

47 

0.67 

0.87  ** 

0.20 

7.  Set  free  text  defaults 

47 

1.20 

1.38 

0.18 

1 .  Verify  platform  role 

53 

0.47 

0.63  * 

0.16 

8.  Create/send  free  text  message 

49 

1.52 

1.67 

0.15 

1 1 .  Create  route  on  map 

46 

2.90 

2.63 

-0.27 

13.  Check  line  of  sight 

49 

1.90 

1.33  ** 

-0.57 

a  Clearing  logs  and  queues  included  time  for  the  FBCB2  workstation  to  reboot.  bThe  recall 
version  of  this  task  included  two  more  steps  than  the  baseline  version. 

*  p  <  .05,  ** p<. 01. 
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The  results  of  the  /-tests  revealed  that  the  baseline-retest  differences  were  significant  for 
nine  of  the  tasks  (eight  cases  increasing  and  one  decreasing),  as  indicated  in  Table  1 6  (/-test 
statistics  appear  in  Appendix  D).  As  Table  16  shows,  average  execution  times  varied 
considerably  across  tasks,  ranging  from  less  than  7  sec  to  nearly  4  min  in  the  baseline  and  from 
25  sec  to  more  than  5  min  in  the  recall  test.  Simple  difference  scores  (recall  test  mean  minus 
baseline  mean)  revealed  that  the  execution  times  increased  by  0.15  min  to  1.47  min  for  11  of  13 
tasks.  This  represented  relative  increases  ranging  from  10%  to  273%.  One  task  exhibited  a 
significant  decrease  in  mean  execution  time  (30%  decline),  and  a  second  task  showed  a  slight 
decrease  (9%  decline). 

Use  of  FBCB2 ’s  Help  Function 

The  FBCB2’s  help  capabilities  were  available  to  the  participants  throughout  the  test  and 
retest  sessions.  The  proctor  neither  discouraged  nor  encouraged  the  students  to  use  the  help 
features  in  performing  the  hands-on  tasks.  Only  during  the  recall  test  were  the  participants  asked 
to  record  whether  they  used  the  help  feature  for  a  given  task.  In  that  session,  27  participants 
(50%  of  the  sample)  reported  using  help  on  at  least  one  task.  Half  of  these  indicated  they  used 
help  more  than  once,  but  no  more  than  five  times.  Those  who  used  help  at  least  once  averaged 
66.4%  correct  in  the  recall  test,  while  those  who  did  not  did  significantly  better  averaging  78.9% 
correct,  /(52)  =  2.13,/?  <  .05. 

Using  help  tended  to  increase  the  total  time  taken  to  perform  all  tasks  in  the  hands-on 
recall  test.  Those  reporting  use  of  help  at  least  once  averaged  37,4  min  overall,  while  those  not 
using  help  averaged  33.1  min.  Although  the  subgroup  using  help  averaged  more  than  4  min 
longer  working  on  the  recall  test,  the  difference  was  not  significant,  /(51)=  1.58,/?=  .12.  The 
participants  were  not  asked  to  indicate  how  much  time  they  spent  using  FBCB2  help  features, 
and  any  increase  in  total  test  time  cannot  be  attributed  solely  to  using  help. 

Table  1 7  shows  the  reported  usage  of  help  functions  for  each  task  in  the  recall  test.  In 
only  three  tasks  did  more  than  10%  of  the  54  participants  report  using  help:  (a)  clear  logs  and 
queues,  (b)  position  icon  on  map,  and  (c)  create  an  address  group.  In  the  case  of  clearing  logs 
and  queues,  3 1%  of  the  participants  reported  using  help.  As  can  be  seen  in  Table  1 7,  few 
participants  used  the  help  function  on  any  given  task  with  the  exception  of  task  #2(17 
individuals).  Interestingly,  only  1 8%  of  the  sample  performed  this  task  correctly  on  the  recall 
test,  the  lowest  percentage  of  all  13  tasks. 

The  reader  should  bear  in  mind  that  the  participants’  responses  regarding  use  of  help 
functions  were  not  independently  verified.  When  a  participant  failed  to  record  “Yes”  or  “No”  for 
using  help  on  a  task,  a  definitive  response  could  not  be  inferred.  Further,  the  participants  were 
not  asked  to  indicate  if  using  help  occurred  before  or  during  task  execution,  nor  how  much  time 
they  spent  using  help.  Finally,  it  cannot  be  assumed  that  resorting  to  help  produced  the 
procedural  information  that  a  participant  w'as  seeking  on  a  particular  task. 
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Table  17 

Self-Reported  Usage  of  System  Help  in  Retest  by  Task 


Task 

#  Who  Reported  Using  Help 

1 .  Verify  platform  role 

2 

2.  Clear  logs  and  queues 

17 

3.  Position  icon  on  map 

7 

4.  Adjust  SA  settings 

3 

5.  Create  message  folders 

2 

6.  Create  address  group 

7 

7.  Set  free  text  defaults 

2 

8.  Create/send  free  text  message 

0 

9.  Create/send  SPOT  report 

3 

10.  Save  incoming  message 

0 

1 1 .  Create  route  on  map 

3 

12.  Center  icon  on  map 

4 

13.  Check  line  of  sight 

0 

Predictors  of  Knowledge  and  Hands-on  Test  Scores 

Correlation  analyses.  The  relationship  between  previous  training  and  experience  and 
performance  on  the  knowledge  and  hands-on  tests  was  examined.  The  analysis  was  performed 
by  computing  Pearson  correlations  between  self-reports  of  training  and  experience  and  scores  on 
the  knowledge  and  hands-on  tests.  The  training  and  experience  measures  included  (a)  self-rated 
FBCB2  proficiency,  (b)  hours  of  FBCB2  operator  training,  (c)  total  collective  FBCB2  training 
experiences,  (d)  months  of  using  FBCB2  in  combat,  and  (e)  composite  computer  experience.  As 
Table  1 8  shows,  collective  training  experience  with  FBCB2  was  the  best  overall  predictor  of 
performance  on  the  knowledge  test.  Similarly,  self-rated  proficiency  was  a  good  predictor  of 
scores  on  the  knowledge  test.  Total  hours  of  FBCB2  training  correlated  significantly  with  the 
total  number  of  items  correct  on  the  baseline,  but  not  the  retest. 

Table  18 

Correlations  between  Training  and  Experience  and  Knowledge  and  Hands-on  Test  Measures 


Component  Score 

Task  Score 

Baseline  Recall 

Baseline  Recall 

Knowledge  Test 

Total  hours  of  FBCB2  operator  training  .3 1  * 

.20 

.22 

.06 

Total  collective  FBCB2  training  experiences 

.38** 

.31  * 

.41  ** 

.38  ** 

Months  using  FBCB2  in  combat 

.1! 

.03 

.14 

.04 

Self-rated  FBCB2  proficiency 

.39  ** 

.31  * 

.35  ** 

.25 

Composite  computer  experience 

.23 

.18 

.24 

.05 
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Table  18  (continued) 


Component  Score 

Task  Score 

Baseline  Recall 

Baseline  Recall 

Hands-on  Test 

Total  hours  of  FBCB2  operator  training 

.12 

.09 

.10 

.03 

Total  collective  FBCB2  training  experiences 

.14 

.25 

.16 

.25 

Months  using  FBCB2  in  combat 

-.15 

-.01 

-.11 

-.09 

Self-rated  FBCB2  proficiency 

.16 

.34* 

.27 

.25 

Composite  computer  experience 

.37  ** 

.38  ** 

.38** 

.43  ** 

*p  <  .05,  **  /?  <  .01. 


Interestingly,  the  only  significant  predictor  of  the  hands-on  test  scores  was  composite 
computer  experience.  In  contrast  to  the  knowledge  test,  previous  individual  and  collective 
training  were  not  related  to  hands-on  performance,  although  self-rated  FBCB2  proficiency 
correlated  significantly  with  the  component  score  in  the  hands-on  recall  test. 


The  correlations  between  the  knowledge  scores  and  the  hands-on  scores  revealed  a 
number  of  significant,  positive  relationships  as  seen  in  Table  19.  In  general,  the  highest 
correlations  occurred  between  recall  scores  on  these  two  tests,  accounting  for  16%  to  28%  of  the 
variance. 


Table  19 

Correlations  between  Knowledge  and  Hands-on  Test  Scores 


Hands-on  Test  Score 

Knowledge  Test  Score 

Component  Score 

Question  Score 

Baseline 

Recall 

Baseline 

Recall 

Component  (baseline) 

.27* 

.27 

.25 

.22 

Component  (recall) 

.4!  ** 

.53  ** 

.41  ** 

.42  ** 

Task  (baseline) 

.32* 

.28* 

.29  * 

.23 

Task  (recall) 

.37  ** 

.51  ** 

.40  ** 

.41  ** 

*p  <  .05,  **  p<  .01. 


Multiple  regression  analyses.  In  the  unit,  it  would  be  relatively  easy  to  collect  general 
information  about  training  history,  experience  and  knowledge  from  individual  operators  of 
FBCB2  in  order  to  predict  skill  decay  over  time.  Accordingly,  the  experience  and  knowledge 
measures  taken  in  this  experiment  were  entered  into  a  series  of  multiple  regression  equations  to 
see  if  they  would  collectively  predict  performance  variability  on  the  hands-on  test  belter  than  the 
knowledge  test  scores  alone  do.  Total  hours  of  FBCB2  training,  the  sum  of  collective  FBCB2 
training  experience,  total  months  of  FBCB2  use  in  combat,  self-rated  FBCB2  proficiency, 
composite  computer  experience,  and  component  score  on  the  baseline  knowledge  test  were  all 
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entered  into  a  standard  regression  model.  Table  20  shows  the  multiple  correlation  coefficients 
for  the  hands-on  test  scores.  As  can  be  seen,  the  regression  coefficients  were  significant  for  three 
measures  of  hands-on  performance — component  recall  score,  baseline  task  score,  and  recall  task 

score. 


Table  20 

Multiple  Correlation  Coefficients  (R)  for  Baseline  and  Recall  Hands-on  Test  Results 


Component  Score 

Task  Score 

Baseline 

Recall 

Baseline 

Recall 

.45 

.56  ** 

.50* 

.57** 

*p  <  .05,  **p<.0\ 


Comparing  the  correlation  coefficients  in  Table  20  to  those  in  Tables  18  and  19,  it  can  be 
seen  that  combining  predictor  variables  improved  the  predictive  model.  For  example,  the 
multiple  correlation  coefficient  for  the  recall  component  score  was  .56  but  the  individual 
predictors  ranged  from  .03  to  .41 .  The  regression  coefficient  for  the  baseline  task  score  was  .50 
and  the  individual  predictors  ranged  from  .10  to  .32.  Finally  the  regression  coefficient  for  the 
recall  task  score  was  .57  while  the  individual  predictor  coefficients  ranged  from  .03  to  .43. 

The  effect  of  one  final  predictor  variable  was  examined — task  difficulty.  To  index  task 
difficulty,  the  number  of  steps  needed  to  complete  each  task  in  the  hands-on  test  was  determined 
with  the  help  of  SMEs.  The  number  of  steps  ranged  from  2  to  22  in  the  baseline  test  and  from  2 
to  24  in  the  recall  test  (see  Appendix  E  for  a  list  of  all  steps).  For  both  tests,  the  median  number 
of  steps  was  7  and  the  modal  number  was  5.  Correlations  between  the  number  of  steps  for  each 
task  and  the  average  score  on  that  task  for  both  the  baseline  and  recall  tests  were  calculated  as  a 
way  to  examine  the  relationship  between  number  of  steps  and  performance.  As  expected,  these 
correlations  were  all  negative  indicating  that  as  the  number  of  steps  increased,  the  average  score 
on  that  task  decreased.  Correlation  coefficients  ranged  from  r  =  -.28  to  r  =  -.42,  but  none 
reached  statistical  significance. 

FBCB2  self-rated  proficiency  groupings.  In  the  previous  section,  the  degree  to  which 
self-rated  FBCB2  proficiency  (on  a  3-point  scale)  was  linearly  correlated  with  the  hands-on  and 
knowledge  test  scores  was  examined.  Those  results  indicated  some  small  but  significant  positive 
correlations  with  knowledge  and  hands-on  test  performance  indicating  that  higher  self-ratings  of 
proficiency  were  associated  with  better  performance  on  the  two  tests.  What  these  correlations 
don't  reveal  is  whether  participants  in  the  three  self-rated  proficiency  groups  differed  on  their  test 
scores.  To  address  this  question,  self-proficiency  ratings  were  used  as  a  grouping  variable  for 
analyses  of  variance  (ANOVA)  to  compare  the  never  used,  basic,  and  medium  proficiency 
groups  on  their  knowledge  and  hands-on  test  scores. 

It  is  important  to  note  that  the  sample  sizes  for  this  analysis  are  not  equal.  To  avoid  the 
confounding  effects  of  unequal  sample  sizes,  ANOVAs  were  performed  using  type  III  sums  of 
squares  which  rely  on  unweighted  means.  Because  only  6  participants  reported  they  had  never 
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used  FBCB2,  while  32  rated  themselves  at  a  basic  level  of  proficiency  and  16  rated  themselves  at 
a  medium  level,  this  type  of  analysis  gave  much  more  weight  to  the  groups  with  no  experience 
and  medium  proficiency  than  the  correlation  analyses  in  the  previous  section. 

Four  outcome  measures  were  analyzed  as  dependent  measures,  while  self-rated  FBCB2 
proficiency  (no  experience,  basic  proficiency,  and  medium  proficiency)  served  as  a  between 
subjects  factor  and  test  (baseline  vs.  recall)  served  as  a  within  subjects  measure.  The  four 
outcome  measures  were  the  component  and  question/task  scores  for  the  knowledge  and  hands-on 
tests.  The  average  scores  for  each  group  appear  in  Figure  2.  In  all  four  analyses,  there  was  a 
significant  main  effect  for  self-rated  proficiency  with  the  never-use  group  scoring  significantly 
lower  than  the  medium  and  low  proficiency  groups  (knowledge  question  score,  F[2,51]  =  4.8,  p 
=  .01 ;  knowledge  component  score,  7^2,51  ]  =  5.1tjp  =  .01 ;  hands-on  task  score,  /|2,51]  =  3.7,  p 
=  .03;  and  hands-on  component  score,  f{2,5l]  =4.0 ,p  =  .03). 


Figure  2.  Mean  scores  on  the  knowledge  and  hands-on  tests  by  self-rated  proficiency  level. 
Panel  A.  knowledge  test  question  scores;  panel  B,  knowledge  test  component  scores;  panel  C, 
hands-on  test  task  scores;  panel  D,  hands-on  test  component  scores, 
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There  were  main  effects  of  test  for  the  hands-on  scores  but  not  the  knowledge  test  scores. 
Both  hands-on  measures  indicated  significant  decay  over  the  retention  interval  (hands-on  task 
score,  F[  1,5 1]  =  19.7,/?  <  .001;  hands-on  component  score,  F[l,51]  =  10.2,/?  =  .002,  see  Figure 
2C  and  2D).  In  addition,  the  analysis  of  the  component  scores  yielded  a  significant  interaction, 
FQ.,5 1 )  =  3.6,  p  =  .04.  The  significant  interaction  resulted  from  the  fact  that  only  the  never-use 
group  exhibited  significant  decay  over  the  retention  interval  (see  Figure  2D). 

As  can  be  seen  in  this  series  of  ANOVAs,  those  individuals  rating  themselves  at  a  basic 
or  medium  level  of  proficiency  were  indistinguishable  on  the  four  outcome  measures.  I  hose 
who  had  never  used  FBCB2  before  fared  significantly  worse— showing  lower  levels  of 
performance  across  the  board,  as  well  as  significant  forgetting  in  their  component  scores  for  the 
hands-on  test  (see  Figure  2D). 


Discussion 

Retention  of  Digital  Operating  Skills 

Over  the  eight  week  retention  interval,  significant  decay  occurred  in  overall  performance 
on  the  hands-on  test  but  not  the  knowledge  test.  More  specifically,  performance  declined 
significantly  on  3  of  the  1 3  hands-on  tasks.  Although  there  was  no  significant  change  in  the 
overall  scores  on  the  knowledge  test,  the  percent  of  the  sample  giving  correct  answers  declined 
significantly  on  one  of  the  questions  and  improved  significantly  on  two  others. 

Predicting  performance  on  the  hands-on  and  knowledge  tests  was  only  moderately 
successful.  Pearson  correlations  between  background  and  experience  variables  and  performance 
on  the  knowledge  and  hands-on  tests  did  not  generally  yield  high  correlations.  Most  variables 
accounted  for  16%  to  25%  of  the  variance.  Combining  all  predictors  in  a  regression  equation  to 
predict  hands-on  test  performance  improved  predictability  slightly  with  multiple  R“  accounting 
for  25%  to  32%  of  the  variance. 

The  greater  forgetting  of  discrete  procedural  tasks  over  explicit  knowledge  is  consistent 
with  evidence  that  declarative  knowledge  is  relatively  resistant  to  decay  even  after  decades  (e.g., 
Bahrick,  1979)  whereas  discrete  procedural  skills  are  highly  perishable  after  only  a  tew  weeks 
(e.g.,  Schendel  et  a).,  1 978).  The  greater  resistance  to  decay  for  declarative  knowledge  seen  in 
the  present  report  has  to  be  considered  in  light  of  the  fact  that  baseline  performance  on  the 
knowledge  task  was  weaker  (average  score  of  40%)  than  on  the  hands-on  test  (average  score  oi 
72%).  The  lower  average  scores  on  the  knowledge  test  diminished  the  potential  for  decay  by 
creating  a  floor  effect. 

The  fact  that  two  of  the  knowledge  test  items  showed  significant  improvement  over  the 
eight  week  retention  interval  must  also  be  factored  in  when  considering  the  apparent  decay 
resistance  of  explicit  knowledge.  There  are  several  possible  explanations  for  these  unexpected 
increases  in  performance  the  most  likely  of  which  is  that  coincidental  learning  took  place  during 
the  retention  interval  (from  classroom  activities  or  discussions  with  colleagues).  Although  the 
participants  reported  that  they  had  not  operated  the  FBCB2  in  the  interim,  it  is  reasonable  to 
expect  that  they  would  have  had  opportunities  to  discuss  FBCB2 -related  topics.  The  fact  that 
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performance  never  improved  significantly  on  any  of  the  hands-on  tasks,  but  did  improve  on  two 
of  the  nine  knowledge  test  questions,  fits  well  with  this  hypothesis. 

Another  possibility  is  that  those  items  showing  improvement  reflected  a  failure  of 
retrieval  on  the  baseline  test.  The  students  may  have  had  lower  motivation  or  perhaps  were  more 
distracted  or  stressed  on  the  baseline  test  than  on  the  recall  test,  It  is  also  possible  that  with  more 
time  to  think  about  and  consolidate  the  information,  participants  were  better  able  to  retrieve  the 
information  during  the  recall  test.  Reminiscence  is  an  established  phenomenon  that  improves 
performance  over  time  (Underwood,  1966),  and  it  is  typically  associated  with  verbal  materials. 

It  is  important  to  keep  in  mind  that  the  questions  on  the  knowledge  test  were  not  meant  to 
serve  as  a  diagnostic  test  for  comprehension  of  the  FBCB2  system.  Although  the  knowledge 
questions  were  designed  by  an  SME,  they  were  only  selected  to  represent  some  of  the  salient 
topics  covered  in  the  two  day  familiarization  training.  The  relatively  low  scores  on  the 
knowledge  test,  even  at  baseline,  cannot  be  interpreted  to  mean  that  the  participants  in  this  class 
had  a  poor  understanding  of  the  FBCB2  system.  Similarly,  high  scores  would  not  necessarily 
indicate  that  an  individual  had  a  strong  grasp  of  the  FBCB2  system. 

The  overall  decline  in  proficiency  on  the  hands-on  test  in  the  present  research  was  1 0% 
for  the  task  scores  and  5%  for  the  component  scores  (both  declines  were  statistically  significant). 
By  comparison,  in  the  investigation  of  IVIS  skills  by  Sanders  (1999),  decay  over  a  30-day  no¬ 
training  period  was  considerably  larger  (23%  for  message  skills  and  52%  for  overlay  skills). 
Although  future  research  will  be  needed  to  definitively  explain  these  discrepant  results,  there 
were  differences  between  the  participants  and  methodologies  that  might  explain  them. 

One  possible  explanation  has  to  do  with  differences  in  the  system -specific  experience  of 
the  participants  in  two  experiments.  Almost  72%  of  the  sample  in  the  present  experiment 
reported  using  FBCB2  during  deployment  in  Iraq  or  Afghanistan.  The  duration  of  usage  in 
combat  typically  ranged  from  3-10  months  but  occasionally  reached  as  high  as  15  months. 

Nearly  a  third  of  the  individuals  rated  their  FBCB2  proficiency  at  a  medium  level.  By 
comparison,  none  of  Sanders’  participants  reported  previous  training  on  IVIS.  It  may  have  been 
that  frequent  use  of  FBCB2  during  combat  deployment  enabled  the  current  experiment’s 
participants  to  form  durable  memories  for  specific  procedures.  It  is  even  possible  that 
participating  in  the  baseline  test  served  to  refresh  knowledge  and  skills  acquired  earlier,  yielding 
a  reactivation  effect  that  could  benefit  performance  in  the  recall  test.  Indeed,  in  the  present 
experiment,  participants  who  indicated  that  they  had  no  prior  experience  using  FBCB2  scored 
significantly  worse  on  the  hands-on  test  and  in  the  case  of  the  hands-on  component  score,  forgot 
significantly  more  than  those  who  rated  themselves  at  a  basic  or  medium  level  of  proficiency. 

In  a  related  vein,  participants  in  the  current  experiment  indicated  substantial  general 
computer  experience.  Nearly  all  individuals  (94%)  reported  using  software  applications  in  a 
Windows  environment,  and  most  (81%)  had  installed  application  software.  In  contrast,  fewer 
than  half  of  Sanders’  participants  (39%)  had  used  a  computer  for  more  than  a  year,  and  one  in 
four  said  they  did  not  use  a  computer  at  all.  The  more  experienced  participants  in  the  present 
research  may  have  been  able  to  apply  their  computer  skills  to  some  degree  when  performing 
FBCB2  tasks.  This  could  have  had  the  effect  of  counteracting  decay  effects.  Composite 
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computer  experience  in  this  research  did  correlate  significantly  with  hands-on  performance 
scores  In  addition  to  having  less  general  computer  experience,  the  participants  in  the  Sanders 
study  were  all  junior  enlisted  Soldiers  (grades  E2-E6).  Participants  in  the  current  experiment 
were  officers  (Captains). 

Methodological  differences  between  Sanders’  investigation  and  the  current  experiment 
may  have  contributed  to  apparent  differences  in  findings  as  well.  Sanders  limited  his  hands-on 
testing  to  overlay  and  message  skills.  On  the  other  hand,  the  present  research  tested  tasks  related 
to  map,  message,  information  management,  navigation,  and  housekeeping  functions.  No  overlay 
tasks  were  included,  but  one  of  the  map  tasks — creating  a  route — is  analogous.  In  terms  ot 
measurement,  Sanders’  primary  retention  measures  were  based  on  a  subset  of  participants  who 
met  a  performance  criterion  on  the  baseline  test.  When  percent  correct  scores  of  his  entire 
sample  were  examined,  they  showed  no  significant  decay  of  message  skills— the  same  as  found 
in  the  present  experiment.  Regarding  map-related  tasks,  the  current  research  found  20%  or 
greater  decline  in  two  map-based  skills,  which  is  more  in  line  with  Sanders  results  for  overlay 
(map-based)  skills. 

The  fact  that  Sanders  used  only  participants  who  performed  to  criterion  (i.e.,  those  who 
demonstrated  they  were  proficient  on  all  tasks  tested)  may  also  account  for  the  higher  rates  of 
decay  observed  in  that  study.  Essentially,  his  participants  were  trained  to  criterion  so  that  they 
were  performing  at  a  high  level  of  proficiency  at  baseline.  In  the  present  experiment,  all 
participants  received  the  same  training  because  the  goal  of  the  course  was  only  to  familiarize 
them  with  the  system.  Statistical  regression  towards  the  mean  alone  would  predict  greater  levels 
of  decay  in  a  group  performing  at  a  higher  than  average  level  of  proficiency  than  one  performing 

at  a  lower  level. 

Another  possible  explanation  relates  to  the  use  of  the  FBCB2  help  function.  Although 
participants  were  allowed  to  use  the  system  help  function  during  both  hands-on  tests,  they  were 
asked  to  record  their  use  of  the  help  function  only  during  the  recall  test.  If  the  self-recording 
process  in  the  retest  encouraged  some  students  to  use  help,  it  could  have  worked  against 
detecting  decay  effects.  This  explanation  does  not  seem  likely,  however,  because  using  help  was 
associated  with  lower  overall  performance— in  particular,  significantly  lower  recall  test  scores 
and  non-significantly  longer  execution  times.  Thus  it  seems  more  likely  that  using  help  was  a 
sign  of  uncertainty  about  how  to  perform  certain  tasks,  with  assistance  from  help  features  tailing 
to  improve  performance.  Nevertheless,  it  is  possible  that  without  the  help  function,  those  who 
used  it  would  have  done  even  worse  and  the  average  levels  of  skill  decay  would  have  increased. 
On  balance,  closer  examination  of  type  of  task  and  type  of  measure  indicates  the  results  of  this 
experiment  are  more  similar  to  Sanders’  findings  than  it  first  appears. 

A  less  direct  measure  of  skill  decay— the  time  spent  completing  the  hands-on  tasks— also 
indicated  a  decline  in  proficiency.  The  total  time  taken  to  perform  all  1 3  tasks  (reading  and 
processing  instructions  plus  executing  the  steps)  increased  from  an  average  of  28  min  in  the 
baseline  test  to  35  min  in  the  recall  test  (25%  increase).  This  measure  correlated  at  low,  non¬ 
significant  levels  with  accuracy  measures  of  performance  (coefficients  smaller  than  ,25). 

Average  execution  times  for  each  task  on  the  baseline  test  ranged  from  6  sec  on  the  low  end  to 
nearly  4  min  on  the  high  end,  and  from  25  sec  to  more  than  5  min  on  the  recall  test.  There  are  no 
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standards  to  compare  these  time  values  against,  and  the  experimenters  did  not  use  SMEs  to 
obtain  empirically  based  estimates  of  expert  performance  times.  It  is  worth  noting  that  Sanders 
(1999),  following  a  30-day  decay  interval,  also  found  significant  increases  in  performance  time 
for  IVIS  overlay  and  report  tasks. 

A  slower  rate  of  executing  procedural  tasks  after  a  substantial  period  of  non-use  is 
unsurprising,  but  a  faster  rate  after  a  decay  period  has  no  ready  explanation.  The  significant 
decrease  in  execution  time  occurred  for  checking  the  line  of  sight,  where  task  accuracy  scores 
increased  slightly  but  non-si gni ft cantly.  No  participants  reported  using  help  for  this  task  in  the 
recall  test.  Two  factors  could  have  been  at  work  to  produce  the  faster  rate  of  execution.  First, 
coincidental  learning  during  the  eight-week  retention  interval  may  have  dealt,  in  part,  with  the 
line  of  sight  function.  This  is  supported  by  the  fact  that  performance  on  the  knowledge  test 
question  regarding  the  line  of  sight  tool  increased  significantly  in  the  recall  test.  Second,  prior 
experience  using  FBCB2  may  have  yielded  a  specific  advantage  for  applying  the  line  of  sight 
junction,  with  the  baseline  test  reactivating  the  procedural  memory. 

Predictors  of  Test  Performance 

Knowledge  and  experience.  Correlation  analyses  were  used  to  determine  how  well 
knowledge  and  hands-on  test  scores  could  be  predicted.  In  general,  baseline  and  recall  test 
scores  correlated  significantly  and  positively  with  each  other  for  the  hands-on  and  knowledge 
tests.  Weaker,  yet  still  significant,  positive  correlations  also  existed  across  the  hands-on  and 
knowledge  tests. 

Other  potential  knowledge  and  experience  predictors  of  performance  were  examined  for 
both  the  knowledge  and  hands-on  tests.  Somewhat  surprisingly,  significant  predictors  of  scores 
on  these  two  types  of  test  showed  minimal  overlap.  Total  collective  FBCB2  training  experiences 
and  self-rated  FBCB2  proficiency  were  the  best,  although  moderate,  predictors  of  performance 
on  the  knowledge  test.  In  contrast,  composite  computer  experience  was  the  best  predictor  of 
performance  on  the  hands-on  test.  The  total  collective  training  experiences  did  not  correlate 
significantly  with  any  of  the  measures  of  hands-on  performance  and  self-rated  FBCB2 
proficiency  correlated  significantly  with  only  the  recall  component  score.  Months  of  use  of 
FBCB2  in  combat  did  not  significantly  correlate  with  measures  on  either  the  hands-on  or 
knowledge  test. 

The  fact  that  overall  computer  experience  was  the  best  single,  although  moderate, 
predictor  of  performance  on  the  hands-on  test  suggests  that  there  is  something  about  experience 
using  personal  computers  that  positively  transfers  to  using  FBCB2.  The  FBCB2  software 
currently  runs  on  a  Unix  operating  system  and  the  graphic  user  interface  is  similar  to  what  is 
found  in  Windows  or  Macintosh  systems.  Thus  there  may  simply  be  positive  transfer  from  these 
systems  when  learning  to  use  FBCB2.  Alternatively,  high  levels  of  general  computer  experience 
may  identify  individuals  who  have  an  aptitude  for  learning  to  use  computers.  The  fact  that  this 
measure  was  a  better  predictor  overall  than  the  total  hours  of  training  on  FBCB2,  the  sum  of 
collective  FBCB2  training  experiences,  the  months  of  FBCB2  use  in  combat  and  the  self-rated 
FBCB2  proficiency  needs  to  be  verified  with  another  sample  of  Soldiers. 
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Task  characteristics.  On  the  knowledge  test,  memorizing  facts  with  low  meaning  or 
connectedness  made  for  a  difficult  learning  challenge.  A  good  example  of  hard-to-memorize 
information  was  found  in  the  names  of  the  four  main  areas  of  the  FBCB2’s  operations  screen. 
These  names  (classification  banner,  map  area,  operations  function  bar,  and  FIPR  bar)  have  low 
intrinsic  meaning  and  don’t  relate  to  terminology  from  a  common  system  such  as  Microsoft 
Windows.  Recall  of  these  names  reached  only  22%  at  the  end  of  the  two-day  course.  Another 
example  was  the  size  limit  for  messages  sent  in  packet  mode,  which  exhibited  65%  decay  on 
average.  These  findings  fit  with  the  literature  on  verbal  learning,  where  low  meaningful  ness  of 
the  subject  material  is  well  known  to  impede  learning  (Underwood,  1966). 

The  number  of  elements  in  a  question  did  not  relate  significantly  to  performance  on  the 
knowledge  test.  The  same  was  true  for  the  number  of  steps  in  a  hands-on  task.  This  was 
somewhat  surprising,  in  light  of  the  literature  relating  task  complexity  to  digital  skill  retention 
(Goodwin,  2006).  It  is  possible  that  the  number  of  elements  or  steps  did  not  provide  a  true  index 
of  complexity  or  a  related  dimension.  It  is  also  possible  that  other  factors,  such  as  chaining  of 
steps  or  system  cuing,  overshadowed  potential  influences  of  number  of  steps.  It  may  be  also  that 
the  experimental  design  of  this  investigation  did  not  set  robust  conditions  for  illuminating  the 
role  of  task  complexity  because  task  complexity  may  have  diminished  performance  at  both 
baseline  and  recall.  For  example,  create  a  SPOT  report,  the  task  with  the  most  steps,  was  only 
performed  by  22%  at  baseline  and  24%  at  recall,  thus  a  failure  to  detect  skill  decay  may  have 
been  hindered  by  a  floor  effect.  In  summary,  there  were  very  few  task  characteristics  that 
adequately  predicted  performance  on  either  the  hands-on  or  knowledge  tests. 

Looking  at  the  hands-on  task  components  that  had  greater  than  1 0%  change  helps  to 
highlight  some  of  the  more  notable  recall  problems.  In  task  3,  placing  the  icon  on  the  map  and 
doing  so  at  a  specific  location  showed  28%  and  13%  decay,  respectively.  Ordinarily  with  GPS, 
this  task  would  be  unnecessary  as  the  system  would  automatically  position  the  vehicle  icon  on 
the  map  at  the  correct  coordinates.  It  is  possible  that,  because  there  was  seldom  a  need  to 
perform  this  function  in  theater,  it  was  more  easily  forgotten.  Another  factor  contributing  to 
decay  may  be  that  the  first  step  to  placing  your  own  vehicle  icon  on  the  map  is  to  press  the  F6 
Admin  button,  which  is  less  intuitive  than  the  FI  Map  button,  for  example,  might  be. 

Another  map  component  task  that  showed  greater  than  1 0%  decay  was  leaving  the  route 
displayed  on  the  map.  The  steps  required  to  perform  this  step  are  not  entirely  intuitive  although 
the  question  provided  a  hint  with  the  phrase  “leave  the  driver’s  display  on."  This  prompt 
resembled  the  checkbox  labeled  “Driver’s  display  on"  that  needed  to  be  checked  to  accomplish 
this  task.  Additional  research  is  needed  to  understand  why  this  was  forgotten. 

The  most  common  messaging  problem  stemmed  from  a  quirk  of  the  address  book  search 
function.  As  explained  in  the  Results  section,  typing  an  addressee  in  the  search  field  does  not 
always  yield  an  exact  match  to  the  right  addressee,  even  when  an  exact  match  is  in  the  address 
book.  It  is  necessary  for  the  user  to  carefully  check  the  addressee  that  is  returned  by  the  search 
function  before  selecting  it.  Failing  to  check  the  addressee  was  a  major  contributor  to  errors  on 
two  of  the  component  tasks  (creating  an  address  group  and  setting  free  text  message  defaults). 
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Another  feature  of  this  system  that  may  have  contributed  to  errors  has  to  do  with  the 
message  options  for  the  SPOT  report.  This  dialogue  box  has  two  tabs,  one  to  set  the  message 
precedence  and  one  to  set  the  message  addressee.  Clicking  on  either  tab  brings  it  to  the 
foreground  and  settings  can  be  changed  for  that  tab  in  a  way  that  is  analogous  to  a  Windows 
system.  At  the  bottom  of  the  message  options  dialogue  box  are  several  buttons  visible  no  matter 
which  tab  is  in  the  foreground.  Those  include  “okay",  “apply”,  and  “close”  buttons.  In  a 
Windows  system,  changes  can  usually  be  made  to  any  tabs  in  any  order  and  then  all  that  is 
needed  to  accept  those  changes  is  to  select  “apply”  or  “okay”,  but  this  is  not  the  way  it  works 
with  FBCB2.  The  “apply”  button  must  be  selected  after  changes  are  made  to  each  tab.  If 
changes  are  made  to  one  tab  and  then  another  is  selected  before  clicking  “apply”,  all  the  changes 
made  to  the  tab  will  revert  to  their  default  settings.  It  is  possible  that  the  large  number  of  errors 
related  to  setting  the  precedence  for  the  SPOT  report  is  due  to  this  idiosyncrasy. 

Next  to  sending  a  SPOT  report,  clearing  logs  and  queues  was  performed  by  the  lowest 
percentage  of  individuals  on  the  baseline  test  and  had  the  lowest  rate  of  all  tasks  on  the  recall 
test.  This  task  had  only  five  steps  but  screen  prompts  for  this  task  are  largely  missing. 
Performing  this  task  required  the  user  to  first  choose  the  “Start”  button  in  the  lower  left  comer  of 
the  screen,  then  “FBCB2”  and  then  “Clear  logs  and  queues”,  followed  by  selecting  items  to  clear 
and  then  selecting  the  “Apply”  button. 

Techniques  for  Counteracting  Decay 

The  results  of  the  present  experiment  point  to  some  ways  current  trainers  might  improve 
training  and  retention  for  FBCB2  operator  skills.  First,  because  we  found  significant  effects  of 
self-rated  proficiency  on  learning  and  recall,  it  could  be  beneficial  to  identify  those  individuals 
who  have  had  no  prior  exposure  to  FBCB2  for  some  remedial  training.  Those  individuals 
indicating  they  had  never  used  FBCB2  learned  less  and  forgot  it  faster  than  those  who  gave 
themselves  low  or  medium  self-ratings  of  proficiency. 

Accommodating  experience  and  proficiency  differences  in  a  single  group  of  students  can 
be  challenging,  but  doing  so  is  important  for  getting  everyone  to  a  higher  level  of  proficiency 
(Leibrecht,  Wampler,  Goodwin,  &  Dyer,  in  preparation;  Wampler  et  al.,  2006).  Those  with  no 
previous  exposure  to  the  system  may  be  especially  motivated  to  learn,  a  factor  upon  which 
instructors  can  capitalize.  As  Leibrecht  et  al.  (in  preparation)  noted,  the  digitally  savvy  students 
can  buddy  with  less  experienced  colleagues  to  serve  as  demonstrators  and  discussion  group 
participants.  The  inexperienced  students  can  be  encouraged  to  seek  help  from  the  instructors  or 
peer  coaches,  both  during  and  after  a  training  session. 

It  was  also  clear  that  some  topics  and  tasks  were  easily  forgotten  by  the  students.  Thus, 
developing  mnemonics  or  other  memory  aids — or  perhaps  reallocating  time  to  focus  on  some  of 
the  more  easily  forgotten  topics  and  tasks — could  help  counteract  their  tendency  to  be  forgotten. 
For  example,  instructors  might  develop  mnemonics  for  the  names  of  the  two  main  FBCB2 
screens,  the  largest  allowable  message  size,  the  four  main  areas  of  the  operations  screen,  and  the 
advantages  of  FBCB2  and  BFT.  Similarly,  greater  emphasis  needs  to  be  given  to  helping 
students  to  remember  the  steps  for  clearing  logs  and  queues,  creating  a  SPOT  report,  selecting 
addressees  using  the  search  function,  and  manually  placing  their  icon  on  the  map. 
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Still  other  training  emphasis  might  be  added  to  the  use  of  the  FBCB2  help  function.  As 
reported  above,  use  of  the  help  system  was  associated  with  lower  success  on  the  hands-on  tasks. 
Although  this  most  likely  indicates  that  those  using  the  help  function  were  least  knowledgeable 
about  the  system,  one  might  expect  that  the  help  function  would  have  been  more  effective  at 
compensating  for  those  deficiencies. 

Lessons  Learned 

Because  the  participants  were  taking  time  out  of  the  Captain’s  Career  Course  to 
participate  in  this  data  collection  effort,  it  was  important  to  keep  the  time  and  scheduling 
complexity  to  a  minimum  by  testing  all  participants  simultaneously  T  his  constraint  required  us 
to  make  some  trade-offs  so  that  we  could  collect  a  large  amount  of  data  m  a  short  period  of  time. 

One  trade-off  was  our  limited  ability  to  measure  and  evaluate  process  during  the  hands- 
on  test  In  other  words,  we  could  not  record  each  individual  menu  choice  or  movement  of  t  e 
pointing  device,  a  constraint  that  reduced  our  ability  to  understand  some  of  the  sources  of  error 
leading  to  forgetting.  For  example,  to  determine  whether  participants  successhilly  cleared  logs 
and  queues,  a  message  was  sent  to  all  systems  prior  to  the  test.  Participants  who  successfully 
cleared  logs  and  queues  eliminated  the  message.  This  allowed  us  to  assign  a  go/no-go  to  each 
student  for  that  task,  but  it  did  not  allow  us  to  determine  exactly  which  step  of  that  task  was  most 
likely  to  be  forgotten.  On  most  of  the  other  tasks,  however,  there  were  multiple  footprints  left  on 
the  system  that  did  allow  us  to  more  precisely  pinpoint  the  source  of  error  For  example  for  the 
task  create,  save,  and  send  a  SPOT  report”  there  were  12  separate  items  that  could  be  confirmed 

on  each  workstation. 

We  feel  that  accepting  this  trade-off  was  reasonable  for  a  couple  of  reasons.  First  of  all, 
bv  choosing  a  series  of  relatively  basic  tasks  for  the  hands-on  test,  it  was  always  possible  to 
narrow  the  source  of  errors  down  to  a  few  steps.  Even  for  a  task  like  clearing  logs  and  queues, 
which  had  only  five  steps,  we  can  safely  assume  that  the  error  probably  „ 

among  the  first  two  steps  as  the  last  three  steps  involved  selecting  apply  close  ,  and  close. 

],  is  hard  to  imagine  that  someone  who  got  past  the  first  two  steps  would  have  been  unable  to 
complete  the  last  three.  Secondly,  we  never  viewed  this  research  effort  as  providing  tht  final 
answers  to  all  questions  about  the  decay  of  FBCB2  skills.  Instead,  we  realized  that  .t  would  help 
us  to  focus  future  research  efforts  and  in  that  regard  it  has  been  successful. 

Another  trade-off  was  choosing  to  observe  a  sample  of  convenience.  The  ICCC  students 
only  received  two  days  of  orientation  training.  The  training  was  intended  to  familiarize  new 
users  with  the  system's  capabilities  and  to  provide  some  refresher  training  to  individuals  with 
prior  experience  on  the  system.  There  was  no  proficiency  requirement  for  this  course  and  as  a 
result  baseline  proficiency  measures  were  less  than  perfect,  sometimes  substantially  so.  The 
primary  disadvantage  of  using  this  sample  was  that  it  was  not  optimal  for  tracking  skill  decay. 
Nevertheless,  this  trade-off  was  deemed  acceptable  because  we  wanted  to  test  our  design 
data  collection  techniques  on  a  readily  available  sample  before  going  to  the  expense  of  a  larger 
scTdrXtion  effort  with  a  sample  that  would  be  considerably  more  difficult  to  track  for  a 
retention  test.  Furthermore,  it  could  be  argued  that  the  proficiency  levels  of  this  sample  may 
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more  closely  resemble  those  observed  in  a  typical  unit  (i.e.,  it’s  unlikely  that  any  unit  ever  has 
proficiency  in  1 00%  of  its  members)  and  so  in  that  regard  these  skill  decay  data  provide  insight 
into  the  rate  of  decay  that  might  be  expected  in  actual  units. 

Using  participants’  self-reported  start  and  stop  times  and  use  of  the  help  system  was 
another  trade-off  that  allowed  us  to  perform  the  data  collection  en  masse  Based  on  the  results,  it 
appears  that  very  few  participants  recorded  obviously  flawed  times  suggesting  this  was  an 
acceptable  means  of  recording  this  measure.  The  accuracy  with  which  participants  recorded 
their  own  use  of  the  help  system  is  more  difficult  to  assess.  Half  of  all  participants  reported 
using  the  help  function  at  least  once  and  every  effort  was  made  by  the  experimenters  to  indicate 
that  using  the  help  function  was  acceptable. 

The  fact  that  we  only  asked  participants  to  record  their  use  of  help  on  the  recall  test  may 
have  produced  a  minor  confounding  in  our  results.  There  may  have  been  an  implicit  assumption 
by  many  of  the  participants  that  using  help  was  not  allowed  during  the  baseline  test  since  no 
instructions  were  given  regarding  its  use.  If  more  participants  used  help  during  the  recall  test  as 
a  result  of  being  asked  to  indicate  when  they  did  so,  this  may  have  inflated  retention  scores. 
Arguing  against  this  possibility  is  the  fact  that  those  who  used  help  at  least  once  did  significantly 
worse  on  the  recall  test  than  those  who  did  not.  Stili,  in  future  studies,  using  the  help  function 
should  be  recorded  during  all  hands-on  test  sessions. 

Conclusions 

For  the  FBCB2  skills  investigated  in  this  report,  significant  decay  was  observed  over  an 
eight  week  retention  interval.  The  good  news  is  that  overall  performance  declined  by  only  10%. 
In  fact,  there  was  significant  decay  on  only  three  of  the  13  tasks,  each  with  19-24%  decrement. 
When  looking  at  the  components  of  those  tasks,  it  appears  that  most  of  the  decay  was  associated 
with  forgetting  how  to  place  an  icon  on  the  map,  forgetting  how  to  leave  a  route  displayed,  and 
forgetting  how  to  assign  the  correct  addressee.  The  elapsed  time  measure  appears  to  be  a  more 
sensitive  measure  of  decay,  showing  significant  skill  decline  on  eight  of  the  1 3  tasks  and 
improvement  for  one  task. 

Understanding  the  causes  of  the  decay  on  these  tasks  is  still  speculative,  but  it  is  unlikely 
that  a  single  factor  accounts  for  all  forgetting.  The  background  experience  and  training  of 
individuals  contributes  to  decay,  predicting  about  25-35%  of  performance  variability.  Vague  or 
absent  screen  prompts,  system  idiosyncrasies,  and  an  inability  to  use  the  system  help  effectively 
also  account  for  some  decay,  but  a  precise  number  can't  be  attributed  to  those  factors  just  yet. 

These  findings  identify  potentially  vulnerable  FBCB2  skills  and  suggest  that  trainers  may 
want  to  devote  additional  instructional  time  or  emphasis  to  these  topics.  It  is  important  to  keep 
in  mind  that  this  is  the  first  empirical  analysis  of  FBCB2  skill  decay.  Future  studies  will  need  to 
be  done  to  confirm  these  results  before  a  strong  recommendation  can  be  made  to  alter  training 
approaches  or  programs  of  instruction.  Furthermore,  future  research  on  FBCB2  skill  decay 
should  include  a  wider  range  of  tasks  that  represent  operational  uses  of  the  system,  and  should 
focus  on  understanding  the  exact  causes  of  forgetting.  This  will  help  system  designers  and 
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training  developers  to  make  informed  decisions  about  how  to  modify  the  system  and  the  way 
Soldiers  should  train  to  use  it. 
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APPENDIX  A 
ACRONYMS 


A-l 


ABCS 

AFATDS 

ANOVA 

AS3 

ASAS 

BFT 

BIOS 

C3 

CPX 

CTC 

FBCB2 

FDO 

FIPR 

FSO 

FTX 

GPS 

ICCC 

IVIS 

KB 

LOS 

MCS 

NAV 

NET 

OA 

OBC 

S3 

S6 

SA 

SEM 

SME 

TOC 

TRADOC 


Army  Battle  Command  System 

Advanced  Field  Artillery  Tactical  Data  System 

analysis  of  variance 

Assistant  Operations  Officer 

All  Source  Analysis  System 

Blue  Force  Tracking 

Basic  Input  and  Output  System 

computerized  command  and  control 
command  post  exercise 
combat  training  center 

Force  XXI  Battle  Command  Brigade  and  Below 

fire  direction  officer 

flash,  immediate,  priority,  routine 

fire  support  officer 

field  training  exercise 

Global  Positioning  System 

Infantry  Captains  Career  Course 
Inter-Vehicular  Information  System 

kilobyte 

line  of  sight 

Maneuver  Control  System 
navigation 

new  equipment  training 

operator  acknowledgment 
Officer  Basic  Course 

operations  officer 
information  management  officer 
situational  awareness 
standard  error  of  the  mean 
subject  matter  expert 

tactical  operations  center 

U.S.  Army  Training  and  Doctrine  Command 
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APPENDIX  B 

KNOWLEDGE  AND  HANDS-ON  MEASUREMENT  INSTRUMENTS 
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FBCB2  Training  and  Experience 
Baseline  Measurement 


The  purpose  of  this  research  project  is  to  understand  which  FBCB2  knowledge  and  skills  are 
most  quickly  forgotten  and  therefore  are  in  need  of  improved  initial  and/or  sustainment  training. 
In  addition  to  this  baseline  measurement,  you  will  take  part  in  an  8-week  follow  up  assessment. 

Your  responses  to  all  questions  are  used  for  research  purposes  only  and  will  be  kept  confidential. 
Nobody  will  have  access  to  them  outside  of  the  research  team  and  your  score  will  not  impact 
your  standing  in  the  ICCC  course. 

So  that  we  can  compare  your  responses  now  and  at  8  weeks,  please  provide  your  last  name  and 
first  initial. 


Last  Name,  First  Initial 


PRIVACY  ACT  STATEMENT 

Public  Law  93-573,  called  the  Privacy  Act  of  1975,  requires  that  you  be  informed  of 
the  purpose  and  uses  to  be  made  of  the  information  collected. 

The  Department  of  the  Army  may  collect  the  information  requested  in  this  survey 
under  the  authority  of  10  United  States  Code  2358.  Providing  information  is 
voluntary.  Failure  to  respond  to  any  particular  questions  will  not  result  in  any 
penalty. 

The  information  collected  in  the  survey  will  be  used  solely  for  research  purposes. 
Your  responses  will  be  held  in  strict  confidence.  No  one  outside  the  research  team 
will  have  access  to  individual  data. 


Do  Not  Turn  This  Page  Until  Told  To  Do  So 
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Experience  and  Training  History  with  Army  Battle  Command  Systems 


1 .  Prior  to  attending  ICCC,  tell  us  how  many  hours  and  the  date  of  your  individual  system 
operator  training  for  each  of  the  digital  systems  listed  below. 


Training 

FBCB2 

ASAS 

MCS 

AFATDS 

No  Operator  Training  (check) 

□ 

□ 

□ 

□ 

Online  Course 

Hours 

Date(year) 

Trained  in  OBC 

Hours 

Date(year) 

NET  Training 

Hours 

Date(year) 

NET  Delta  Training  (trained 

Hours 

on  changes  and  upgrades) 

Date(year) 

Digital  Master  Trainer  Course 

Hours 

Date(year) 

Other: 

Hours 

Date(year) 

2.  Indicate  the  types  of  collective  unit  training  you  have  had  with  the  following  systems,  prior 
to  attending  ICCC.  Place  an  “X”  in  all  appropriate  boxes. 


FBCB2 

ASAS 

MCS 

AFATDS 

No  collective  unit  training  Received 

Company  or  Platoon  Motorpool  training 

Company  or  Platoon  FTX  at  home  station 

Company  or  Platoon  FTX  at  a  CTC 

CPX  training  in  a  digital  training  facility 

CPX  training  at  home  station 

CPX  training  at  a  CTC 

Other: 

Next  Page 
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3.  Have  you  ever  used  any  digital  systems  while  deployed  on  a  combat  tour?  d)YES  d)NO 


If  you  answered  YES,  complete  the  table  below.  If  you  answered  NO  go  on  to  question  4. 


System  and 
Version 

Combat  Theater 

Duty  position  when  using  this 
system 

Number  of  months 
you  used  this 
system  in  theater 

4.  Overall  how  would  you  rate  your  proficiency  on  each  system? 

Basic  -  You  can  use  the  system  to  perform  a  limited  set  of  functions  but  there  are 
many  aspects  of  the  system  with  which  you  are  unfamiliar. 

Medium  -  You  are  comfortable  with  the  system  and  are  knowledgeable  about  most  of  its 
functions  and  quirks.  You  have  limited  troubleshooting  abilities. 

High  —  You  have  advanced  knowledge  of  this  system  and  can  troubleshoot  many 

problems.  You  frequently  are  asked  to  help  others  who  have  difficulty  w'ith 
the  system. 

Check  the  appropriate  boxes: 


System 

Never 

Used 

Basic 

Medium 

High 

FBCB2 

□ 

□ 

□ 

□ 

ASAS 

□ 

□ 

□ 

□ 

MCS 

□ 

□ 

□ 

□ 

AFATDS 

□ 

□ 

□ 

□ 
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5.  Indicate  the  type  of  experience  you  have  using  the  following  operating  systems  (check  all  that 
apply): 


Task 

Windows 

Mac  OS 

Linux 

Never  Used  this  operating  system 

□ 

□ 

□ 

Used  application  software  (e.g..  Outlook,  PowerPoint, 
games) 

□ 

□ 

□ 

Installed  application  software 

□ 

□ 

□ 

Installed  software  patches 

□ 

□ 

□ 

Installed  Hardware  (e.g.,  hard  drive,  graphics  card) 

□ 

□ 

□ 

Changed  boot-up  options 

□ 

□ 

□ 

Changed  BIOS  settings 

□ 

n 

□ 

Authored  web  pages  using  software  on  this  system 

□ 

□ 

□ 

Authored  programs  for  this  operating  system 

□ 

□ 

□ 

FBCB2  Training  and  Experience 
Recall  Measurement 


1 .  Have  you  operated  FBCB2  for  any  reason  since  the  baseline  test  we  administered  on  24  May? 
(Circle  one):  YES  NO 

If  you  answered  NO,  Proceed  to  the  next  page. 

2.  If  you  answered  YES,  estimate  the  total  time  (in  hours)  you  have  used  FBCB2  since  taking 
the  baseline  test  on  24  May. 

Estimate  of  hours:  _ 
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FBCB2  KNOWLEDGE  TEST 
BASELINE  AND  RECALL  MEASUREMENTS 

INSTRUCTIONS:  Complete  the  following  questions  to  the  best  of  your  ability. 

1 .  What  does  FBCB2  stand  for? 

2.  What  does  the  acronym  FIPR  stand  for? 

a.  F:  _ 

b.  I:  _ 

c.  P:  _ 

d.  R:  _ 

3.  Name  the  2  screens  on  the  FBCB2  system. 

a.  _ 

b.  _ 

4.  What  are  the  four  main  areas  of  the  Operations  screen? 

a. _ 


5.  The  largest  message  size  sent  in  Packet  Mode  on  FBCB2  is _ bytes. 

a.  576 

b.  800 

c.  1450 

d.  1280 


6.  What  is  meant  by  a  “Near  Real  Time  Update”? _ 

7.  Besides  the  dead  space,  the  Line  of  Sight  Tool  allows  the  Soldier  to  quickly  determine  the 

and _ _  .  between  two  points  selected  on  the  map 

screen. 

8.  If  your  system  is  running  unusually  slow,  what  can  you  do  to  speed  it  up? _ 

9.  List  one  advantage  of  FBCB2-TerrestriaI  and  FBCB2-BFT 

FBCB2-Terrestrial: _ _ __ _ _ _ _ _ 

FBCB2-BFT: _ _ _ 

]  0.  When  assigning  a  periodic  reminder,  which  time  zone  must  you  enter  the  time  in? 
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HANDS-ON  TEST  -  FBCB2  OPERATOR  TASKS 
BASELINE  MEASUREMENT 


INSTRUCTIONS: 

Using  the  FBCB2  system,  execute  the  13  FBCB2  operator  tasks  on  the  following  pages  to  the 
best  of  your  ability.  If  you  cannot  complete  any  task,  indicate  that  you  could  not  perform  the 
task  and  move  on  to  the  next  task,  Proctors  and  Instructors  will  not  assist  you  in  answering  any 
of  the  questions. 

For  each  of  the  1 3  tasks,  do  the  following: 

1.  READ  THE  TASK. 

2.  RECORD  YOUR  START  TIME.  Record  the  exact  time  in  terms  of  hour,  minute,  and 
seconds,  (e.g.,  1 1 :33:21) 

3.  RECORD  THE  STOP  TIME  FOR  THE  TASK  YOU  JUST  COMPLETED. 

4.  CONTINUE  WITH  THE  NEXT  TASK. 

Once  you  have  completed  all  tasks  raise  your  hand  and  someone  will  come  by  and  check  your 
sheets  and  release  you.  Thank  you  for  your  cooperation  and  assistance  in  this  research  effort. 


PROCEED  AT  YOUR  OWN  PACE. 


Next  Page 

BE  SURE  TO  RECORD 

YOUR  START 

AND 

STOP 

TIMES!!!! 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESHI 


Task  1.  What  is  the  role  your  computer  is  set  to? 

Task  1  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Enter  the  role  here _ . 

Task  1  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  2.  Clear  ALL  your  logs  and  queues  and  ensure  your  map  is  displayed  for  operations. 

Task  2  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  2  STOP  TIME  _ _ : _ 

(hh)  (mm)  (ss) 


Task  3.  Position  your  icon  at  grid  loc  vie  14RPV  21184  50414. 

Task  3  START  TIME _ : _ : 

(hh)  (mm)  (ss) 

Task  3  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  4.  Set  your  SA  settings  to  Stale:  12  hours;  Old:  24  hours;  and  Purge:  48  hours. 

Task  4  START  TIME _ _ : _ 

(hh)  (mm)  (ss) 

Task  4  STOP  TIME  : _ : _ 

(hh)  (mm)  (ss) 


Next  page 

BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESHI 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 


Task  5.  Create  two  message  folders.  Name  one  ‘company messages'  and  the  other 
‘companyoverlays’. 

Task  5  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  5  STOP  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  6.  Create  an  address  group  named  TOC.  The  TOC  address  you  will  use  is  cdr-bde4- 
3id. 

Task  6  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  6  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  7.  Create  a  default  message  address  for  free  text  messages.  Precedence  is 
immediate,  acknowledge  is  operator  acknowledge,  and  the  unit  address  you  are  sending  the 
message  to  is  cdr-bde4-3id. 

Task  7  START  TIME _ : _ : _ 


(hh)  (mm)  (ss) 


Task  7  STOP  TIME 


(hh)  (mm)  (ss) 


Next  Page 

BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 


Task  8.  Create  and  send  a  free  text  message  with  the  subject  line  as  READY.  Text  should 
read  as  follows:  IN  POSITION  AND  READY  TO  EXECUTE  MISSION.  Set  precedence  to 

priority  and  send  it  to  the  TOC  address  group. 

Task  8  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  8  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  9.  Create,  Save,  and  send  a  SPOT  report  to  the  Brigade  Commander,  4lh  Bde,  3d  ID, 
You  have  just  spotted  an  armored  personnel  carrier  and  8  dismounted  personnel,  at  grid 
location  14RPV  18515  49816.  The  affiliation  for  the  target  is  suspect,  and  it  is  assembling. 
Precedence  should  be  set  to  Immediate  and  acknowledgement  set  to  OA.  Save  the  message 
as  SPOT1.  Send  the  message. 

Task  9  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  9  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  10.  Display  and  save  a  message.  You  have  received  a  message  from  the  CDR  4lh 
Brigade.  Display  the  message,  save  it  as  MOVE  I,  and  save  it  in  company  messages. 

Task  10  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  10  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 


Task  11.  Create  a  route  with  5  waypoints  -  the  first  should  be  the  start  point  (SP)  which  is 
your  present  location.  The  last  waypoint  should  be  the  release  point  (RP).  Name  the  route 
ROUTE0O01.  Use  the  map  to  select  your  waypoints.  Be  sure  to  leave  the  driver  display 
ON. 

Task  11  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  11  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  12.  Auto  center  your  icon. 

Task  12  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  12  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  13.  Create  a  line  of  sight  (LOS)  from  your  own  location  to  14RPV  18514  49839.  Set 
the  height  above  ground  as  2  meters. 

LEAVE  THE  LOS  WINDOW  DISPLAYED. 

Task  13  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Task  13  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 

You  have  completed  the  test.  Raise  your  hand  and  a  proctor  will  check  your  test  and  release  you. 
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HANDS-ON  TEST  -  FBCB2  OPERATOR  TASKS 
RECALL  MEASUREMENT 


INSTRUCTIONS: 

Using  the  FBCB2  system,  execute  the  13  FBCB2  operator  tasks  on  the  following  pages  to  the 
best  of  your  ability.  If  you  cannot  complete  any  task,  indicate  that  you  could  not  perform  the 
task  and  move  on  to  the  next  task.  Proctors  and  Instructors  will  not  assist  you  in  answering  any 
of  the  questions. 

For  each  of  the  13  tasks,  do  the  following: 

1.  READ  THE  TASK. 

2.  RECORD  YOUR  START  TIME.  Record  the  exact  time  in  terms  of  hour,  minute,  and 
seconds,  (e.g.,  11:33:21) 

3.  RECORD  THE  STOP  TIME  FOR  THE  TASK  YOU  JUST  COMPLETED. 

4.  CONTINUE  WITH  THE  NEXT  TASK. 

5.  If  you  do  NOT  KNOW  HOW  to  complete  a  task,  write  in  the  area  for  that  task, 
“UNABLE  TO  COMPLETE”  and  move  on  to  the  next  task. 

6.  If  you  used  the  Help  for  that  task,  please  indicate  so  by  checking  the  appropriate  box  for 
each  question. 

Once  you  have  completed  all  tasks  raise  your  hand  and  someone  will  come  by  and  check  your 
sheets  and  release  you.  Thank  you  for  your  cooperation  and  assistance  in  this  research  effort. 

PROCEED  AT  YOUR  OWN  PACE. 

Next  Page 

BE  SURE  TO  RECORD 

YOUR  START 

AND 

STOP  TIMESIIII 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 


Task  1 .  What  is  the  role  your  computer  is  set  to? 

Task  1  START  TIME _ : _ : _ 

(hh)  (mm)  ( ss ) 

Enter  the  role  here _ . 

Did  you  use  the  help  function?  Yes  O  No  D 

Task  1  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 

Task  2.  Clear  ALL  your  logs  and  queues  and  ensure  your  map  is  displayed  for  operations. 

Task  2  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  □  No  D 

Task  2  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 

Task  3.  Position  your  icon  at  grid  loc  vie  14RPV  20397  48244. 

Task  3  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  D  No  CH 

Task  3  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESI!! 


Task  4.  Set  your  SA  settings  to  Stale:  6  hours;  Old:  12  hours;  and  Purge:  36  hours. 

Task  4  START  TIME _ : _ _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  Id  No  Id 

Task  4  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  5.  Create  two  message  folders.  Name  one  ‘unitmessages’  and  the  other  ‘unitoverlays*. 

Task  5  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  d  No  □ 

Task  5  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 

Task  6.  Create  an  address  group  named  TOC.  The  TOC  address  you  will  use  is  cdr-bdel- 
3id 

Task  6  START  TIME  _ _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  d  No  d 

Task  6  STOP  TIME  _ _ : _ 

(hh)  (mm)  (ss) 

Next  Page 

BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 


Task  7.  Create  a  default  message  address  for  free  text  messages.  Precedence  is 
immediate,  acknowledge  is  operator  acknowledge,  and  the  unit  address  you  are  sending  the 
message  to  is  cdr-bdcl-3id. 

Task  7  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  C]  No  D 

Task  7  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  8.  Create  and  send  a  free  text  message  with  the  subject  line  as  PROBLEM.  Text 
should  read  as  follows:  B26  HAS  LOST  POWER.  Set  precedence  to  priority  and  send  it  to 
the  TOC  address  group. 

Task  8  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  Q  No  □ 

Task  8  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Next  Page 

BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESHI 


Task  9.  Create,  Save,  and  send  a  SPOT  report  to  the  Brigade  Commander,  lsl  Bde,  3d  ID. 
You  have  just  spotted  three  attack  helicopters,  one  with  6  anti-tank  missiles,  at  grid 
location  14RPV  1652  5053,  The  affiliation  for  the  target  is  hostile,  and  it  is  reconnoitering. 
They  are  moving  at  medium  speed,  heading  southeast.  Precedence  should  be  set  to 
Immediate,  and  acknowledgement  set  to  OA.  Save  the  message  as  SPOT6.  Send  the 
message. 

Task  9  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  CH  No  □ 

Task  9  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  10.  Display  and  save  a  message.  You  have  received  a  message  from  the  CDR  !sl 
Brigade.  Display  the  message,  save  it  as  MOVE4,  and  save  it  in  unitmessages. 

Task  10  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  CH  No  HU 

Task  10  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  II.  Create  a  route  with  5  waypoints  -  the  first  should  be  the  start  point  (SP)  which  is 
your  present  location.  The  last  waypoint  should  be  the  release  point  (RP).  Name  the  route 
ROUTEPUMA.  Use  the  map  to  select  your  wavpoints.  Be  sure  to  leave  the  driver  displav 
ON. 


Task  11  START  TIME 


(hh) 

(mm) 

(ss) 

Did  you  use  the  help  function? 

Yes  □ 

No  □ 

Task  11  STOP  TIME 

*  * 

4  * 

(hh) 

(mm) 

(ss) 

Next  Page 

BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESI11 
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BE  SURE  TO  RECORD  YOUR  START  AND  STOP  TIMESIII 


Task  12.  Auto  center  your  icon. 

Task  12  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  □  No  □ 

Task  12  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


Task  13.  Create  a  line  of  Sight  (LOS)  from  the  last  location  you  sighted  the  enemy 
helicopters  to  your  location.  Set  the  height  above  ground  as  50  meters. 

LEAVE  THE  LOS  WINDOW  DISPLAYED. 

Task  13  START  TIME _ : _ : _ 

(hh)  (mm)  (ss) 

Did  you  use  the  help  function?  Yes  HH  No  dl 

Task  13  STOP  TIME  _ : _ : _ 

(hh)  (mm)  (ss) 


You  have  completed  the  test.  Raise  your  hand  and  a  proctor  will  check  your  test  and  release  you. 
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APPENDIX  C 

SCORING  MATERIALS  FOR  KNOWLEDGE  AND  HANDS-ON  TESTS 
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FBCB2  Knowledge  Test  -  Scoring  Rules 

NOTE-1:  Ignore  spelling  errors  and  transposition  errors  throughout. 
NOTE-2:  Ignore  student  indications  of  guessing  when  answer  is  correct. 


1 .  What  does  FBCB2  stand  for?  {Force  XXI  Battle  Command  Brigade  &  Below) 

Range  of  Scores:  0-5;  credit  one  point  for  each  correct  element  (Force  XXI  -  Battle  -  Command  - 
Brigade  -  &  Below)  regardless  of  sequence;  “Force”  without  “XXI’'  is  OK;  ignore  extra  words  unless 
they  clearly  indicate  guessing  (e.g.,  Future  Force). 

2.  What  does  the  acronym  FIPR  stand  for? 

a.  F:  _ (Flash) 

b.  I;  _ (Immediate) 

c.  P: _ (Priority) 

d.  R:  _ (Routine) 

Range  of  Scores:  0-4;  credit  one  point  for  each  correct  item,  regardless  of  sequence;  ignore  extra 
words  unless  they  clearly  indicate  guessing  (e.g.,  Instant  Immediate). 

3.  Name  the  2  screens  on  the  FBCB2  system. 

a.  _ (Session  Manager  Screen  or  Session  or  Manager  or  Admin  Screen) 

b.  _ (OPS  Screen  or  Operations  Screen  or  Ops  or  Operation  or  Operating 

Screen) 

Range  of  Scores:  0-2;  credit  one  point  for  each  correct  item;  ignore  extra  words  unless  they  clearly 
indicate  guessing. 

4.  What  are  the  four  main  areas  of  the  Operations  screen? 

a.  _ (Classification  banner  or  Classification  bar  or  Classification) 

b.  _ _ (SA/Map  Area  or  SA  or  Map  or  SA  area  or  Map  area  or  Situation 

Awareness) 

c.  _ (Ops  function  bar  or  Ops  bar  or  Ops  function  or  Functions  bar  or  Operations 

bar) 

d.  _ _  (Com  Status/FI  PR  bar  or  Com  Status  or  FIPR  or  FI  PR/Com  or  FIPR  Status) 

Range  of  Scores:  0-4;  credit  one  point  for  each  correct  item,  regardless  of  sequence;  ignore  extra 
words  unless  they  clearly  indicate  guessing. 

5.  The  largest  message  size  sent  in  Packet  Mode  on  FBCB2  is _ bytes.  (C) 

a.  576 

b.  800 

c.  H50 

d.  1280 

Range  of  Scores:  0-1;  one  point  if  answer  is  correct;  if  two  or  more  answers  are  circled,  score  —  0. 
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6.  What  is  meant  by  a  “Near  Real  Time  Update”?  ***  Suspend  this  question  from  scoring 


(Network/bandwidth  limitations  delay  transmission;  or  SA  icons  may  be  inaccurate;  or 
transmission  of  location  information  is  delayed;  or  update  is  triggered  by  time  or  movement 
criteria) 

Range  of  Scores:  Not  Applicable 

7.  Besides  the  dead  space,  the  Line  of  Sight  Tool  allows  the  Soldier  to  quickly  determine  the 

_ and _ ,  between  two  points  selected  on  the  map 

screen. 

(azimuth/bearing/heading/direction)  (distance/range/meters)  (elevation) 

Range  of  Scores:  0-2;  credit  one  point  for  each  correct  item,  regardless  of  sequence:  ignore  extra 
words  unless  they  clearly  indicate  guessing. 

8.  If  your  system  is  running  unusually  slow,  what  can  you  do  to  speed  it  up? 

_ (Clear  Logs  &  Queues  or 

Delete  messages/overlays/files/data  or  Clean  out  folders  or  Free  up  space  or  Reset  system) 

Range  of  Scores:  0-1;  credit  one  point  if  answer  suggests  correct  understanding;  if  answer  merely 
discusses  speed,  score  =  0. 

9.  List  one  advantage  of  FBCB2-Terrestrial  and  FBCB2-BFT 

FBCB2-Terrestrial; _ 

(Larger  bandwidth  or  shows  individual  vehicles  or  secure  network) 

FBCB2  -  BFT _ 

(Terrain  does  not  impede  signal  or  Satellite  link  overcomes  LOS  limit  or  Relay  is  unnecessary) 

Range  of  Scores:  0-2;  credit  one  point  for  each  answer  that  conveys  correct  understanding;  ignore 
extra  verbiage  as  long  as  correct  answer  is  present;  if  answer  states  operating  principle  but  no 
advantage,  score  =  0. 

10.  When  assigning  a  periodic  reminder,  which  time  zone  must  you  enter  the  time  in? _ 

(ZULU  or  GMT  or  Greenwich  Mean  Time) 

Range  of  Scores:  0-1;  one  point  for  correct  answer;  ignore  extra  words  unless  they  clearly  indicate 
guessing. 
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Baseline  Hands-on  Data  Collection  instrument 


Name  of  person  being  graded: 
Name  of  data  collector: _ 


Task  13.  LOS 

A.  Is  the  line  of  sight  profile  showing?  Yes  No 

B.  Is  the  distance  above  ground  set  to  2  meters?  Yes  No 

C.  Comments  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  for  each  Yes;  ignore  comments. 


Task  1.  Role 

A.  Is  the  role  recorded  correctly?  Yes  No 

B.  Comments  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-1;  one  point  if  la  is  Yes;  one  point  if  la  is  No  and  student 
response  is  a  close  approximation  of  the  correct  role. 


Task  3.  Position  Icon  on  the  Map 

A.  Is  the  icon  on  the  map?  Yes  No 

B.  Click  on  the  role  to  see  location 

Is  the  icon  located  precisely  at  14  RPV  21184  50414?  Yes  No 

C.  Comments  (if  any  items  are  “NO"  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  for  each  Yes;  one  point  if  3b  is  No  and  recorded 
coordinates  show  evidence  of  typo(s). 


Task  2.  Clear  All  Logs  and  Queues 

A.  Check  the  FIPR  Queue  -  is  there  more  than  one  message?  Yes  No 
Open  F4  Messages  -  go  to  the  Sent  Queue 

B.  Are  any  of  the  messages  older  than  the  start  time  of  the  test?  Yes  No 

C.  Are  there  any  other  directories  present?  [originally  5c]  Yes  No 

D.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-3;  one  point  if  2a  is  Yes;  “missing  data”  if  2a  is  No  and  FIPR 
queue  contains  zero  messages;  one  point  if  2b  is  No;  one  point  if  2c  is  No. 
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Task  4.  SA  Settings 

Click  F6  Admin.  Select  the  SA  Tab  and  Friendly 

A.  Is  Stale  set  to  12  hrs?  Yes  No 

B.  Is  Old  set  to  24  hours?  Yes  No 

C.  Is  Purge  set  to  48  hours?  Yes  No 

D.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0*3;  one  point  for  each  Yes;  if  “correct”  values  result  from 
failure  to  clear  logs  and  queues,  score  =  0;  if  elapsed  time  is  zero,  score  =  0. 


Task  5.  Create  Message  and  Overlay  folders 
Click  F4  Messages.  Select  the  Manage  Tab. 

A.  Is  there  a  directory  named  companymessages?  Yes  No 

B.  Is  there  a  directory  named  companyoverlays?  Yes  No 

C.  Comments  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  if  5a  is  Yes,  one  point  if  5b  is  Yes;  if  the  target 
directory  names  (“companymessages”  and  “companyoverlays”)  are  missing  but 
close  approximations  are  found,  award  one  point  for  each  (two  points  possible). 


Task  6.  Create  an  Address  Group  Named  TOC 

F4  Messages.  Select  the  Create  Tab.  Select  Edit  Address  Groups. 

A.  Is  there  a  message  address  group  called  TOC?  Yes  No 

B.  Is  the  addressee  cdr-bde1-3id  (4B)?  Yes  No 

C.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  for  each  Yes;  ignore  comments. 


Task  7.  Set  the  free  text  message  default  address 

Select  Create  tab.  Highlight  ‘free  text  message”.  Select  Message  Addressing. 

A.  Is  the  addressee  cdr-bde1-3id?  Yes  No 

B.  Is  the  precedence  set  to  immediate ?  Yes  No 

C.  Is  the  acknowledgement  set  to  operator  acknowledge?  Yes  No 

D.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0*3;  one  point  for  each  Yes;  ignore  comments. 
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Task  8.  Create  and  Send  a  Free  text  message 

Select  Sent  Queue  tab.  Highlight  the  free  text  message.  Select  Execute. 

A.  Was  the  free  text  message  sent?  Yes  No 

B.  Is  the  addressee  the  TOC  address  group?  Yes  No 

C.  Comments:  (if  any  items  are  “NO"  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  for  each  Yes;  one  point  if  8b  is  No  and  address 
matches  the  TOC  address  created  in  Task  6. 


Task  9.  Create,  save,  and  send  a  SPOT  report. 

Select  the  Edit  tab.  Expand  companymessages  and  select  SPOT1.  Click  on 
Execute. 

Scroll  down  to  view  message. 

A.  Was  there  a  message  named  SPOT1  in  companymessages?  Yes  No 

B.  Did  the  message  indicate  an  APC?  Yes  No 

C.  Did  the  message  indicate  the  location  14RPV185  498?  Yes  No 

D.  Did  the  message  indicate  they  were  suspect?  Yes  No 

E.  Did  the  message  indicate  they  were  assembling?  Yes  No 

F.  Did  the  message  indicate  there  were  8  dismounts?  Yes  No 

G.  Was  the  precedence  set  to  immediate ?  Yes  No 

H.  Check  the  Sent  Queue  -  did  it  indicate  that  SPOT1  was  sent?  Yes  No 

I.  Is  the  addressee  cdr-bde1-3id  (4B)?  Yes  No 

J.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-10;  one  point  if  any  Spot  report  is  found  in 
“companymessages”  folder  or  in  “combat  messages”  folder;  one  point  if  report 
name  is  “SPOT1”;  one  point  for  each  Yes  in  9b  thru  9i. 


Task  10.  Display  and  save  a  message. 

Select  the  Manage  tab.  Expand  companymessages. 

A.  Is  there  a  message  saved  that  is  named  MOVE1?  Yes  No 

B.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-1;  one  point  if  10a  is  Yes;  one  point  if  10a  is  No  and  report 
name  is  a  variation  of  MOVE  1. 


NOTE  -  Close  F4  Messages  when  this  task  is  completed. 
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Task  11.  Create  a  Route  with  5  waypoints. 

Select  NAV.  Select  the  down  arrow  and  locate  ROUTEOO01.  Select  it. 

A.  Is  the  route  displayed  on  the  map?  Yes  No 

A.  Is  the  route  name  ROUTEOOOI?  Yes  No 

B.  Indicate  the  number  of  waypoints  in  the  route 


C.  Comments:  (if  any  items  are  “NO"  copy  student's  response  below) 

Range  of  Scores:  0-3;  one  point  if  1  la  is  Yes;  one  point  if  1  lb  is  Yes;  one  point  if 
11c  is  5. 


Task  12.  Center  Icon  on  Map 

A.  Is  the  blue  filled  icon  in  the  center  of  the  map  sheet?  Yes  No 

B.  Is  the  AutoCenter  off?  Yes  No 

C.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-1;  one  point  if  12a  is  Yes;  exclude  12b  from  scoring. 


NOTE-1 :  Ignore  errors  (especially  spelling  and  transposition)  in  students’ 
handwritten  responses. 

NOTE-2:  If  a  clear  answer  cannot  be  determined  for  an  item,  treat  it  as  a  missing 
data  point. 
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Recall  Hands-on  Data  Collection  Instrument 


Name  of  person  being  graded: 
Name  of  data  collector: _ 


Task  13.  LOS 

D.  Is  the  line  of  sight  profile  showing?  Yes  No 

E.  Is  the  distance  above  ground  set  to  50  meters?  Yes  No 

F.  Comments  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  for  each  Yes;  ignore  comments. 


Task  1.  Role 

A.  Is  the  role  recorded  correctly?  Yes  No 

B.  Comments  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-1;  one  point  if  la  is  Yes;  one  point  if  la  is  No  and  student 
response  is  a  close  approximation  of  the  correct  role. 


Task  3.  Position  Icon  on  the  Map 

A.  Is  the  icon  on  the  map?  Yes  No 

B.  Click  on  the  role  to  see  location 

Is  the  icon  located  precisely  at  14  RPV  20397  48244?  Yes  No 

C.  Comments  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  for  each  Yes;  one  point  if  3b  is  No  and  recorded 
coordinates  show  evidence  of  typo(s). 


Task  2.  Clear  All  Logs  and  Queues 

A.  Check  the  FIPR  Queue  -  is  there  only  one  message?  Yes  No 
Open  F4  Messages  -  go  to  the  Sent  Queue 

B.  Are  any  of  the  messages  older  than  the  start  time  of  the  test?  Yes  No 

C.  Are  there  any  other  directories  present?  [originally  5c]  Yes  No 

D.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-3;  one  point  if  2a  is  Yes;  “missing  data”  if  2a  is  No  and  FIPR 
queue  contains  zero  messages;  one  point  if  2b  is  No;  one  point  if  2c  is  No. 
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Task  4.  SA  Settings 

Click  F6  Admin.  Select  the  SA  Tab  and  Friendly 

A.  Is  Stale  set  to  6  hrs?  Yes  No 

B.  Is  Old  set  to  12  hours?  Yes  No 

C.  Is  Purge  set  to  36  hours?  Yes  No 

D.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0*3;  one  point  for  each  Yes;  if  “correct”  values  result  from 
failure  to  clear  logs  and  queues,  score  =  0;  if  elapsed  time  is  zero,  score  =  0. 


Task  5.  Create  Message  and  Overlay  folders 
Click  F4  Messages.  Select  the  Manage  Tab. 

A.  Is  there  a  directory  named  unitmessages?  Yes  No 

B.  Is  there  a  directory  named  unitoverlays?  Yes  No 

C.  Comments  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  if  5a  is  Yes,  one  point  if  5b  is  Yes;  if  the  target 
directory  names  (“unitmessages”  and  “unitoverlays”)  are  missing  but  close 
approximations  are  found,  award  one  point  for  each  (two  points  possible). 


Task  6.  Create  an  Address  Group  Name  TOC 

Click  F4  Messages.  Select  the  Create  Tab.  Select  Edit  Address  Groups. 

A.  Is  there  a  message  address  group  called  TOC?  Yes  No 

B.  Is  the  addressee  cdr-bde1-3id?  Yes  No 

C.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  for  each  Yes;  ignore  comments. 


Task  7.  Set  the  free  text  message  default  address 

Select  Create  tab.  Highlight  ‘free  text  message”.  Select  Message  Addressing. 

A.  Is  the  addressee  cdr-bde1-3id?  Yes  No 
Select  the  Message  Setting  Tab  to  the  left. 

B.  Is  the  precedence  set  to  immediate ?  Yes  No 

C.  Is  the  acknowledgement  set  to  operator  acknowledge ?  Yes  No 

D.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0*3;  one  point  for  each  Yes;  ignore  comments. 
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Task  8.  Create  and  Send  a  Free  text  message 

Select  Sent  Queue  tab.  Highlight  the  free  text  message.  Select  Execute. 

A.  Was  the  free  text  message  sent?  Yes  No 

B.  Is  the  addressee  the  Bde  Cdr?  Yes  No 

C.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-2;  one  point  for  each  Yes;  one  point  if  8b  is  No  and  address 
matches  the  TOC  address  created  in  Task  6. 


Task  9.  Create ,  save,  and  send  a  SPOT  report. 

Select  the  Edit  tab.  Expand  unitmessages  and  select  SPOT1.  Click  Execute. 
Select  the  Preview  Tab.  Scroll  down  to  view  message. 

A.  Was  there  a  message  named  SPOT1  in  unitmessages?  Yes  No**See  Note 

B.  Did  the  message  indicate  3  rotary  wing  aircraft?  Yes  No 

C.  Did  the  message  indicate  the  entity  location  14RPV1 652  5053?  Yes  No 

D.  Did  the  message  indicate  they  were  hostile?  Yes  No 

E.  Did  the  message  indicate  they  were  reconnoitering?  Yes  No 

F.  Did  the  message  indicate  there  were  6  AT  missiles?  Yes  No 

G.  Did  the  message  indicate  they  were  moving  at  medium  speed?  Yes  No 

H.  Did  the  message  indicate  they  were  moving  to  the  southeast?  Yes  No 
Select  the  Message  Options  button  on  the  lower  left  corner  of  the  screen. 
Select  the  Message  Settings  tab 

I.  Was  the  precedence  set  to  immediate ?  Yes  No 

J.  Check  the  Sent  Queue  -  did  it  indicate  that  SPOT1  was  sent?  Yes  No 
Select  the  Addressee  tab. 

K.  Is  the  addressee  cdr-bde3-3id?  Yes  No 

**Note:  If  it  is  not  in  unitmessages,  is  it  in  combatmessages?  Yes  No 
If  it  is  in  combatmessages,  is  it  named  Spotl?  Yes  No 

L.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-12;  one  point  if  any  Spot  report  is  found  in  “unitmessages” 
folder  or  in  “combat  messages”  folder;  one  point  if  report  name  is  “SPOT1”;  one 
point  for  each  Yes  in  9b  thru  9k.  [77ns  item  has  two  more  points  possible  than 
the  same  item  in  the  initial  £es£.] 


Task  10.  Display  and  save  a  message. 

Select  the  Manage  tab.  Expand  unitmessages. 

A.  Is  there  a  message  saved  that  is  named  MOVE1?  Yes  No 

B.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-1;  one  point  if  10a  is  Yes;  one  point  if  10a  is  No  and  report 
name  is  a  variation  of  MOVE4. 

NOTE  -  Close  F4  Messages  when  this  task  is  completed. 
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Task  11.  Create  a  Route  with  5  waypoints. 

Select  NAV.  Select  the  down  arrow  and  locate  ROUTEPUMA.  Select  it. 

A.  Is  the  route  displayed  on  the  map?  Yes  No 

B.  Is  the  route  name  ROUTEPUMA?  Yes  No 

C.  Indicate  the  number  of  waypoints  in  the  route 


D.  Comments:  (if  any  items  are  "NO”  copy  student’s  response  below) 

Range  of  Scores:  0-3;  one  point  if  11a  is  Yes;  one  point  if  lib  is  Yes;  one  point  if 
11c  is  5. 


Task  12.  Center  Icon  on  Map 

A.  Is  the  blue  filled  icon  in  the  center  of  the  map  sheet?  Yes  No 

B.  Is  the  AutoCenter  off?  Yes  No 

C.  Comments:  (if  any  items  are  “NO”  copy  student’s  response  below) 

Range  of  Scores:  0-1;  one  point  if  12a  is  Yes;  exclude  12b  from  scoring. 


NOTE-1 :  Ignore  errors  (especially  spelling  and  transposition)  in  students’ 
handwritten  responses. 

NOTE-2:  If  a  clear  answer  cannot  be  determined  for  an  item,  treat  it  as  a  missing 
data  point. 
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APPENDIX  D 

STATISTICAL  TABLES  FOR  ITEM  ANALYSES 
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Table  D1 

Chi-Square  (McNemar)  Results  for  Knowledge  Test  Questions 


Question 

Improve 

Decline 

x2 

P 

5 

Packet  mode  message  size  limit? 

2 

26 

20.57 

0.000  4 

3 

The  two  system  screens  are? 

1 

7 

4.50 

0.070 

2 

FIPR  stands  for  what? 

3 

7 

1.60 

0.344 

10 

Time  zone  to  enter  reminders? 

7 

9 

0.25 

0.804 

9 

Advantage  of  FBCB2-T  &  BFT? 

0 

0 

0.00 

n.s. 

4 

Four  main  areas  of  Ops  screen? 

0 

0 

0.00 

n.s. 

8 

How  to  speed  up  a  slow  system? 

9 

4 

1.92 

0.267 

7 

LOS  tool  determines  what? 

15 

5 

5.00 

0.041  * 

1 

FBCB2  stands  for  what? 

14 

3 

7.12 

0.013  * 

Note.  The  “Improve”  column  gives  the  number  of  individuals  whose  scores  improved  across  the 
two  tests  and  the  “Decline”  column  lists  the  number  whose  scores  declined  across  the  two  tests. 

*  p 5-05. 
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Table  D2 

Chi-Square  (McNemar)  Results  for  Knowledge  Test  Components 


Question  /  Component 

Improve 

Decline 

l2 

P 

3 

Name  the  two  screens  on  FBCB2. 

Session  manager 

2 

7 

2.78 

0.180 

Operations 

7 

14 

2.33 

0.189 

2 

What  does  FIPR  stand  for? 

Flash 

3 

7 

1.60 

0.344 

Immediate 

3 

5 

0.50 

0.727 

Priority 

5 

6 

0.09 

1.000 

Routine 

4 

5 

0.11 

1.000 

9 

List  one  advantage  to  FBCB2  and  BFT 

FBCB2 

0 

0 

0.00 

n.s. 

BFT 

2 

3 

0.20 

1.000 

4 

What  are  the  four  main  areas  of  the  Ops  screen? 
Classification  banner 

0 

7 

7.00 

0,016 

* 

Map  display 

10 

6 

1.00 

0.454 

Operations  function  bar 

2 

10 

5.33 

0.039 

* 

Communications/FIPR  queue 

2 

4 

0.67 

0.688 

7 

The  LOS  tool  allows  the  Soldier  to  determine? 
Bearing 

14 

5 

4.26 

0.064 

Range 

14 

6 

3.20 

0.115 

1 

What  does  FBCB2  stand  for? 

Force  2 1 

13 

2 

8.07 

0.007 

* 

Battle 

11 

5 

2.25 

0.21 

Command 

10 

5 

1.67 

0.302 

Brigade  and 

12 

2 

7.14 

0.013 

* 

Below 

12 

2 

7.14 

0.013 

* 

Note,  Questions  5,  8,  and  10  are  all  single  component  questions  so  their  McNemar  results  are 
not  repeated  from  Table  D 1 .  *  p  < .05. 
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Table  D3 

Chi-Square  (McNemar)  Results  for  Hands-On  Test  Task  Scores 


Task 

Improve 

Decline 

x2 

P 

12 

Auto  center  icon  on  map 

3 

16 

8.89 

0.004  * 

11 

Create  route  on  map 

5 

16 

5.76 

0.027  * 

6 

Create  address  group 

2 

12 

7.14 

0.013  * 

2 

Clear  logs  and  queues 

5 

13 

3.56 

0.096 

3 

Position  icon  on  map 

4 

11 

3.27 

0.118 

7 

Set  free  text  defaults 

5 

12 

2.88 

0.143 

10 

Save  incoming  message 

4 

11 

3.27 

0.118 

4 

Adjust  SA  settings 

4 

10 

2.57 

0.180 

5 

Create  message  folders 

0 

5 

5.00 

0.063 

I 

Verify  platform  role 

2 

3 

0.20 

1.000 

8 

Create/send  free  text  message 

5 

5 

0.00 

1.000 

9 

Create/send  SPOT  report 

8 

7 

0.07 

1.000 

13 

Check  line  of  sight 

12 

10 

0.18 

0.832 

Note :  The  “Improve”  column  gives  the  number  of  individuals  whose  scores  improved  across  the 
two  tests  and  the  “Decline”  column  lists  the  number  whose  scores  declined  across  the  two  tests. 

*  p<.05. 


D-4 


Table  D4 

Chi-Square  (McNemar)  Results  for  Hands-on  Test  Component  Scores 


Task  /  Component 

Improve 

Decline 

x2 

P 

11 

Create  route  on  map 

Route  displayed  on  screen 

3 

20 

12.57 

0.000  * 

Route  named  correctly 

4 

9 

1.92 

0.267 

Route  has  correct  number  of  waypoints 

6 

10 

1.00 

0.454 

6 

Create  address  group 

Address  group  named 

2 

7 

2.78 

0.180 

Address  group  addressee  correct 

2 

12 

7.14 

0.013  * 

2 

Clear  logs  and  queues 

One  message  in  FIPR  queue 

7 

9 

0.25 

0.804 

No  old  messages 

7 

6 

0.08 

1.000 

No  other  directories 

15 

9 

1.50 

0.307 

3 

Position  icon  on  map 

Icon  on  Map 

0 

15 

15.00 

0.000  * 

Icon  Location  correct 

4 

11 

3.27 

0.118 

7 

Set  free  text  defaults 

Set  free  text  message  default  address 

4 

15 

6.37 

0.019  * 

Set  free  text  message  default  precedence 

7 

8 

0.07 

1. 000 

Set  free  text  message  default  acknowledge 

7 

9 

0.25 

0.804 

4 

Adjust  SA  settings 

SA:  stale  setting  correct 

4 

9 

1.92 

0.267 

SA:  old  setting  correct 

4 

7 

0.82 

0.549 

SA:  purge  setting  correct 

4 

8 

1.33 

0,388 

5 

Create  message  folders 

Directory  1  named 

0 

3 

3.00 

0.250 

Directory  2  named 

0 

5 

5.00 

0.063 

8 

Create/send  free  text  message 

Free  text  message  sent 

3 

4 

0.14 

1.000 

Free  text  message  address  correct 

5 

5 

0.00 

1.000 

9 

Create/send  SPOT  report 

SPOT  report  in  folder 

8 

4 

1.33 

0.388 

SPOT  report  named  correctly 

6 

9 

0.60 

0.607 

SPOT:  vehiclel  indicated 

10 

9 

0.05 

1.000 

SPOT:  entity  1  location  indicated 

13 

3 

6.25 

0.021  * 

SPOT:  entity  1  status  indicated 

8 

5 

0.69 

0.581 

SPOT:  entity  1  activity  indicated 

8 

8 

0.00 

1.000 

SPOT:  entity  2  indicated 

7 

7 

0.00 

1.000 

SPOT:  precedence  indicated 

7 

11 

0.89 

0.481 

SPOT:  report  sent 

6 

13 

2.58 

0.167 

SPOT :  address  correct 

15 

10 

1.00 

0.424 

13 

Check  line  of  sight 

LOS  is  displayed 

12 

11 

0.04 

1. 000 

LOS  height  correct 

10 

7 

0.53 

0.629 

Note:  Tasks  1,10,  and  12  had  only  one  component  so  their  McNemar  results  are  not  repeated 
from  Table  D3. 

*  p  <  .05. 
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Table  D5 

Paired \-Test  Results  (Two-Tailed)  for  Performance  Times  in  Hands-on  Tasks 


Task 

Baseline  Mean 
(min) 

Recall  Mean 
(min) 

/ 

df 

P 

3.  Position  icon  on  map 

1.55 

3.02 

4.13 

47 

<.001 

** 

2.  Clear  logs  and  queues  a 

3.77 

5.07 

3.53 

46 

<001 

** 

6.  Create  address  group 

1.83 

2.57 

3.42 

40 

.001 

*  * 

4.  Adjust  SA  settings 

0.85 

1.48 

5.12 

52 

<001 

** 

9.  Create/send  SPOT  report b 

3.63 

4.17 

1.77 

46 

.083 

5.  Create  message  folders 

1.17 

1.68 

2.85 

52 

.006 

** 

12.  Center  icon  on  map 

0.11 

0.41 

3.82 

44 

<001 

** 

10.  Save  incoming  message 

0.67 

0.87 

3.18 

46 

.003 

** 

7.  Set  free  text  defaults 

1.20 

1.38 

1.47 

46 

.147 

1.  Verify  platform  role 

0.47 

0.63 

2.51 

52 

.015 

* 

8.  Create/send  free  text  message 

1.52 

1.67 

1.16 

48 

.250 

1 1 .  Create  route  on  map 

2.90 

2.63 

0.78 

45 

.439 

13.  Check  line  of  sight 

1.90 

1.33 

2.93 

48 

.005 

** 

a  Clearing  logs  and  queues  included  time  for  the  FBCB2  workstation  to  reboot.  bThe  recall 
version  of  this  task  included  two  more  steps  than  the  baseline  version. 

*  p<  .05, 
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Table  D6 

Means  and  Standard  Errors  of  Performance  on  the  Hands-on  and  Knowledge  Tests  by  Self- 
Rated  Proficiency 


Test 

Never  Use 

Basic 

Medium 

Knowledge  Test  Question  Score 

Baseline 

24%  (±3%) 

41%  (±3%) 

45%  (±3%) 

Recall 

30%  (±5%) 

38%  (±2%) 

42%  (±3%) 

Knowledge  Test  Component  Score 

Baseline 

26%  (±6%) 

46%  (±4%) 

54%  (±3%) 

Recall 

34%  (±7%) 

46%  (±3%) 

52%  (±3%) 

Hands-on  Test  Task  Score 

Baseline 

60%  (±6%) 

71%  (±3%) 

76%  (±3%) 

Recall 

38%  (±9%) 

64%  (±4%) 

64%  (±5%) 

Hands-on  Test  Component  Score 

Baseline 

72%  (±6%) 

78%  (±3%) 

81%  (±3%) 

Recall 

46%  (±11%) 

75%  (±4%) 

78%  (±3%) 
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APPENDIX  E 

STEPS  FOR  COMPLETING  HANDS-ON  TASKS 


E-l 


I .  Task:  Identify  Platform  Role  Configuration 


Steps: 

1.  While  in  the  Session  Manager  screen,  note  the  platform  role  as  listed  in  the  bottom  pane 
of  the  function  bar,  located  on  the  lower  right  hand  side  of  the  screen. 

2.  Write  the  platform  role  on  the  answer  sheet. 

Note:  This  task  does  not  involve  any  manipulation  of  the  user  interface. 


2.  Task:  Clear  Logs  and  Queues 


Steps: 

1 .  While  in  the  Session  Manager  Screen  and  offline,  select  ‘Start’  on  the  task  bar.  Select 
‘FBCB2’  from  the  menu,  and  ‘Clear  Logs  and  Queues’  from  the  FBCB2  sub-menu. 

2.  Select  items  to  be  cleared  by  checking  the  radio  button  next  to  the  desired  item  (note:  the 
‘reset’  radio  button,  located  at  the  bottom  of  the  dialogue  box,  clears  all  fields). 

3.  Select  ‘Apply’  to  clear  selected  item(s). 

4.  Select  ‘Close’  button  when  message  “COMPLETED  CLEAR  LOGS  &  QUEUES 
OPERATION”  is  displayed. 

5.  Select  ‘Close’  in  Clear  Logs  and  Queues  dialog  box. 


3.  Task:  Position  your  Icon 


Steps: 

1 .  From  the  Operations  Main  Screen,  select  “F6  Admin”  from  the  function  bar  on  the  right 
side  of  the  screen. 

2.  If  not  already  there,  go  to  the  ‘Platform’  tab,  and  select  the  ‘Location’  sub-tab  (this  is  the 
default). 

3.  At  the  Location  text  field,  select  the  dropdown  arrow.  Select  ‘KBD’  from  the  dropdown 
list  and  type  in  the  location  grid  coordinates,  or  select  ‘Map’  to  click  on  a  location  on  the 
map. 

4.  Select  ‘Apply’ 

5.  Select  ‘Close’ 


4.  Task:  Set  SA  Currency 


Steps: 

1.  From  the  Operations  Main  Screen,  select  ‘F6  Admin' 

2.  Select  the ‘SA’ tab 

3.  Using  the  ‘Friendly,’  ‘Observed,’  and  ‘Air’  sub-tabs,  set  the  currency  thresholds  for 
each  type  of  SA. 

4.  In  each  sub-tab,  set  the  ‘Stale’  time  threshold  by  selecting  the  dropdown  next  to  the 
defaulted  time.  Highlight  the  desired  length  of  time  for  SA  to  remain  fresh. 
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5.  Set  the  ‘Old’  time  threshold  by  selecting  the  proper  length  of  time  for  icons  to  remain 
stale,  from  that  dropdown. 

6.  Set  the  ‘Purge’  time  threshold  by  selecting  the  time  period  from  the  ‘Purge’ 
dropdown,  indicating  when  the  S A  will  be  terminated. 

7.  Select  ‘Apply’. 

8.  Select ‘Close’. 


5.  Task:  Create  Two  Messages  Folders 


Steps: 

1.  While  in  the  Main  Operations  Screen,  select  the  ‘F4  Messages’  button  from  the  function 
bar  on  the  right  side  of  the  screen 

2.  Select  the  ‘Manage’  tab 

3.  Type  a  folder  name  in  ‘Name’  field  found  on  the  lower  right  of  the  manage  tab. 

4.  Select  the  ‘New  Folder’  button  on  the  left  side  of  the  tab. 

5.  Delete  the  name  displayed  in  the  “Name”  text  field,  and  enter  the  name  of  the  second 
folder. 

6.  Select  the  ‘New  Folder’  button  to  create  the  second  folder. 

7.  Select  ‘Close’. 

Note:  steps  5  and  6  are  a  repetition  of  steps  3  and  4. 


6.  Task:  Create  Address  Groups 


Steps: 

1.  Select  the  ‘F4  Messages’  button 

2.  Select  the  ‘Create’  tab  (which  is  the  default) 

3.  Select  the  ‘Edit  Address  Groups’  button 

4.  On  the  lower,  right  side  of  the  tab,  Enter  the  Address  Group  name  in  the  Name  text  field 

5.  Select  the  ‘New  Group  button’  which  is  located  in  the  center  of  the  2  panes. 

6.  Above  the  left  pane,  use  the  “Select  From”  dropdown  to  select  the  appropriate  address 
group. 

7.  Search  for  address  in  the  database  by  typing  an  address  name  in  the  “Search”  field,  at  the 
bottom  of  the  left  pane,  and  select  ‘Search’. 

8.  From  the  right  pane,  highlight  the  Address  Group  where  the  address  will  be  stored. 

9.  Select  ‘Add  Address’,  located  between  the  two  panes,  to  add  the  address  to  the  newly 
created  Address  Group. 

10.  Select  ‘Apply’  to  accept  the  change,  but  keep  working  in  the  “Edit  Address  Groups” 
dialogue  box.  OR  Select  ‘OK’  to  apply  the  change  and  close  the  dialogue  box. 
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7,  Task:  Create  Default  Options  and  Address  for  Free  Text  Messages 


Steps: 

1 .  Select  *F4  Messages’  button  from  the  function  bar  of  the  Operations  Main  Screen. 

2.  Select  the  ‘Create’  tab. 

3.  Select  one  of  the  Msg  Type  option  buttons,  in  the  upper  left  comer  of  the  tab,  to  find  the 
appropriate  message. 

4.  Highlight  “freetext  message”  from  the  list  in  the  center  pane  of  the  tab. 

5.  Select  the  ‘Set  Default  Message  Addressing’  button  to  access  the  “Message  Addressing” 
dialogue  box. 

6.  From  the  two  tabs  in  the  dialogue  box,  select  the  left,  or  “Message  Settings”  tab. 

7.  From  the  Message  Settings  tab  select  the  Precedence  option  by  clicking  on  the 
appropriate  radio  button. 

8.  Select  one  or  more  Acknowledge  check  boxes. 

9.  Select  the  ‘Apply’  button  prior  to  changing  tabs  to  ensure  changes. 

1 0.  Select  “Addresses”  tab 

1 1 .  Using  the  “Select  From”  dropdown,  highlight  the  appropriate  address  book. 

1 2.  Select  desired  Platform  Roles  from  left  pane  displaying  the  chosen  address  book 

13.  Select  ‘Add’  button  between  the  panes 

14.  Select  ‘Apply’ 

15.  Select  ‘Close’ 


8.  Task:  Create  and  Send  a  Free  Text  Message 


Steps: 

1 .  While  in  the  Ops  main  screen,  select  the  ‘F4  Messages’  button  from  the  Function  Bar  on 
the  right  side  of  the  screen. 

2.  Select  the  ‘Create’  tab. 

3.  Highlight  “Free  Text  Message”  from  the  list  of  messages  in  the  center  pane  of  the  tab. 

4.  Select  the  ‘Execute’  button. 

5.  Highlight  the  ‘Subject’  text  field  and  enter  subject. 

6.  Highlight  the  main  text  box  and  type  the  message. 

7.  If  not  defaulted,  select  the  ‘Message  Option’  button  at  the  bottom  of  the  message. 

8.  In  the  Message  Addressing  dialogue  box,  select  the  “Message  Settings”  tab. 

9.  Select  desired  Precedence  and  Acknowledgement  of  the  message. 

10.  Select  the  ‘Apply’  button  to  accept  the  change. 

1 1 .  Select  the  ‘Address’  lab. 

12.  From  the  appropriate  address  book,  select  desired  addressee,  searching  if  necessary. 

13.  Select  ‘Add’  to  add  the  address  to  the  message.  Repeat  steps  1 1  &  12  as  needed. 

14.  Select  ‘OK’  to  apply  to  changes  and  close  the  dialogue  box. 

1 5.  If  the  message  is  to  be  saved,  select  ‘Save  As'  and  provide  a  file  name  and  folder 
destination. 

16.  From  the  message,  select  the  ‘Send’  button. 

17.  If  an  “Address  Availability”  reminder  appears  to  advise  that  some  receiving  platforms  are 
not  on  the  net,  select  ‘Send’  again. 
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NOTE:  On  both  the  baseline  and  retention  tests,  participants  were  instructed  to  alter  the 
precedence  setting  from  the  default  established  in  Task  7.  They  did  not  have  to  change  the 
Acknowledgement  setting.  The  addressee  is  in  the  TOC  address  group  so  there  was  no  need  for 
them  to  search  the  address  book. 


9.  Task:  Create,  Save,  and  Send  a  SPOT  Report 


Steps: 

1 .  Select  the  ‘F3  Combat  Msgs’  button  from  the  function  bar,  on  the  right  side  of  the  Ops 
Main  Screen  (defaults  to  the  SPOT  Report) 

2.  Under  the  ‘SPOT’  tab  select  the  Equipment  type  from  the  drop-down  menu  of  the  top, 
right  text  field.  If  more  than  one  type  of  equipment  is  to  be  reported,  utilize  the 
‘Equipment  2’  text  field,  directly  below. 

3.  Select  the  dropdown  button  next  to  the  “Location”  text  field.  Select  ‘KBD’  to  type  the 
grid  coordinate  in  the  text  field,  or  use  ‘Map’  to  select  a  location  from  the  map. 

4.  Fill  the  DTG  field  using  the  keypad  (defaults  to  NOW) 

5.  In  the  first  field  on  the  right  side,  select  the  Observed  Affiliation  from  drop  down  box 
(defaults  to  “hostile”) 

6.  Select  the  Activity  from  the  drop-down  menu  (Defaults  to  “stationary'”) 

7.  Select  the  ‘Save’  button  at  the  bottom  of  the  SPOT  report,  to  save  the  report  to  the 
Combat  Messages  folder  in  F4  Messages. 

8.  If  Default  Message  Addressing  was  not  done  prior  to  executing  Combat  Messages,  select 
‘Long-Form’  message  button  at  the  bottom  of  the  report. 

9.  Select  the  ‘Message  Options’  button 

10.  Select  the  ‘Message  Settings’  tab 

1 1 .  Select  the  desired  precedence 

12.  Select  the  desired  Acknowledgement 

13.  Select  the  ‘Addresses’  tab 

14.  Select  desired  unit  from  the  left  hand  panel 

15.  Select  ‘Add’ 

16.  Select  ‘OK’ 

17.  Select  ‘Save  As’ 

1 8.  Type  the  filename  for  the  message  in  the  appropriate  space 

19.  Select  the  folder  the  message  is  to  be  saved  in.  NOTE:  COMBAT  MESSAGES  WILL 
NOT  BE  IN  THE  COMBAT  MESSAGES  USER  FOLDER. 

20.  Select  the  ‘OK’ 

21.  Select  the  ‘Send’  button 

22.  Select  the  ‘Send’  button  again 

NOTE:  Some  participants  chose  the  Save  button  from  the  short  form  and  saved  the 

message  in  the  Combat  Messages  folder.  MAKE  SURE  THE  MESSAGE  ADDRESSING  IS 

DEFAULTED  PRIOR  TO  EXECUTING.  FAILURE  TO  DO  SO  DEFEATS  THE 

PURPOSE  OF  THE  1 1 -SECOND  EXECUTION  RULE  FOR  COMBAT  MESSAGES. 

TAKE  CARE  NOT  TO  DELETE  ADDRESSES  ALREADY  DEFAULTED  BY  FBCB2. 

THESE  ABCS  THREADS  MUST  REMAIN  TO  ENSURE  AN  ACCURATE  COP. 
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10.  Task:  Display  and  Save  a  Message 


Steps: 

1.  From  the  OPS  Main  Screen,  Select  appropriate  FIPR  counter  on  the  FIPR  Queue  located 
in  the  status  bar  at  the  top  of  the  screen 

2.  From  the  open  tab,  select  and  highlight  message  received 

3.  Select  the  ‘Display’  button  at  the  bottom  of  the  tab 

4.  Select  ‘Save  As’ 

5.  From  the  ‘Save  As’  dialogue  box,  select  the  appropriate  folder,  and  assign  the  message  a 
file  name 

6.  Select  ’OK’  then  ‘OK’  to  acknowledge  the  ‘Message  Saved’  banner. 

7.  Select  ‘Close’  at  the  bottom  of  the  message 


1 1 .  Create  a  Route  with  Five  Waypoints. 


Steps: 

1 .  Select  the  ‘NA  V’  button  from  the  Function  Bar  of  the  Ops  Main  Screen. 

2.  From  the  “Route”  tab  select  ‘Create’ 

3.  From  the  “Create  Nav  Route”  dialogue  box,  in  the  “Enter  waypoints  field”,  select  ‘Map’ 

4.  Enter  the  waypoints  by  clicking  on  the  map  along  the  route,  or  by  highlighting  the 
“Location”  text  field  and  typing  in  the  appropriate  grid  coordinates. 

5.  Select  ‘OK’  to  exit  the  “Create  Route”  dialogue  box  and  return  to  the  Navigation 
dialogue  box. 

6.  Check  the  ‘Drivers  Display  ON’  radio  button,  then  ‘Steer  To’  button  at  the  bottom  of  the 
Navigation  tool.  The  drivers  display  will  appear  by  which  you  can  maintain  course  while 
navigating  the  route. 

7.  If  navigation  tool  is  not  to  be  used  but  the  route  must  be  displayed,  select  ‘Close’  from 
the  Drivers  Display. 

NOTE:  Any  time  a  route  is  accessed,  ‘Drivers  Display  On’,  ‘Steer  To’,  and  ‘Close’  must 
be  executed  to  keep  the  route  displayed  after  the  Nav  box  is  closed. 


12.  Task:  Auto  Center  Your  Icon 


Steps 

1.  Select  the  auto-center  button  at  the  top  right  corner  of  the  function  bar  in  the  Ops 
Main  Screen.  The  map  will  center  on  platform  icon. 

2.  Turn  the  auto-center  off  by  selecting  the  button  a  second  time. 

NOTE:  Turning  the  auto-center  off  was  not  a  requirement  of  the  test  but  some  may  have 
been  taught  to  turn  it  off  in  a  moving  vehicle  to  avoid  locking  up  the  system. 


E-6 


13.  Task:  Create  a  LOS  between  Two  Points  and  Display 


Steps: 

] .  Select  the  ‘LOS’  button  from  the  Function  Bar  in  the  Ops  Main  Screen 

2.  Select  the  two  points  on  the  map 

3.  Select ‘OK’ 

4.  In  the  “Dist.  Above  Ground  (m)”  field,  enter  the  appropriate  value 

5.  Select  ‘Profile  Show’ 


DEMONSTRATION  TASK:  Using  Circular  LOS  Tool 


Steps: 

1 .  Select  ‘F7  Apps’  button  from  the  Function  Bar  of  the  Ops  Main  Screen 

2.  Highlight  ‘Circular  Line-of-Sight’ 

3.  Select  ‘Execute’  button 

4.  Enter  location  by  using  the  dropdown  next  to  the  text  field  and  choosing  ‘KBD’  to  type 
coordinate,  ‘Map’  to  select  a  location  on  the  map,  or  ‘Own’  to  use  platform  location  as 
center  ofCLOS. 

5.  In  the  “Dist.  Above  Ground  (m)”  field  enter  the  distance  above  ground  for  CLOS 

6.  Enter  Fan  Range 

7.  Select  ‘Execute’  and  FBCB2  will  draw  the  CLOS 

8.  Select  ‘Close’  to  turn  Circular  LOS  “OFF” 
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